The Findings From U.S. Social Experiments

As explained in Section 5, an unusual characteristic of the empirical literature on active labor market policies is that it includes a relatively large number of both experimental and non-experimental studies. However, because treatment non-participation and control

group substitution are often substantial, the parameter measured in experimental studies is the e¤ect of the “intention to treat” and not the impact of “treatment on the treated.”

Dropping out similarly a-icts nonexperimental studies (Heckman, Smith and Taber, 1998), and contamination bias is the counterpart to control group substitution. Accordingly, although the estimates reported in the experimental literature are usually thought to be di¤erent from that in the nonexperimental literature, it is easy to exaggerate the di¤erences.

Nonetheless, because the estimates reported in both literatures do not adjust for these biases, and because the incidence of the various biases may di¤er between experimental and nonexperimental studies it is likely that di¤erent parameters are estimated in these diverse literatures. For these reasons, we survey the two literatures separately.

Provided the assumptions discussed in Section 5 hold, social experiments yield easily computed and widely understood estimates of the “the intention to treat” on the treat-ments’ outcomes. As shown by Table 10.4, collectively, the U.S. experimental evaluations provide some of compelling evidence that the opportunity to receive these services some-times can improve participants’ employment prospects and that the resources spent on these services can pass a standard cost-bene…t test. The most consistent evidence in this regard is found for adult women.⁹⁶ As shown by Table 10.4, the earnings gains received by adult women assigned to the treatment group are (i) usually modest in size ranging from a few hundred dollars to more than one thousand dollars, annually, (ii) often persist at least for several years without signs of decay, (iii) arise from a variety of intended treatments, and (iv) sometimes appear to be remarkably cost e¤ective, at least before the deadweight costs of taxation, displacement and substitution e¤ects are taken into account. Further, although the opportunity to receive job search assistance appears to be the most cost-e¤ective service in the sense that it has the highest IRR, more expensive WE and training programs result in larger absolute earnings gains.

Because of substantial treatment non-participation and control group substitution, the impact of these services on those who actually received them is generally larger than indi-cated by the experimental estimates reported in Table 10.4. The exceptions are the NSW

96In keeping with the emphasis of U.S. policy on reducing reliance on social assistance, most social ex-periments have tested the impact of employment and training services on individuals who were applying for or receiving social assistance or welfare (AFDC). The number of these experiments proliferated during the 1980s after the federal government authorized states to operate as demonstration projects community work experience programs (CWEP) for their welfare population. In several states, o¢cials implemented an experimental design in a few welfare o¢ces by mandating that only a random sample of the eligible population participate in JSA, CWEP, or other employment related activities (Goldman, et al., 1986).

Because the vast majority of social assistance recipients are single female household heads, this has meant that most of the experimental evidence relates to economically disadvantaged adult women. These ex-perimental results were in‡uential in shaping U.S. welfare policy during the late 1980s (Greenberg and Wiseman, 1992).

and AFDC Homemaker-Health Care demonstrations. As explained in Section 5, the NSW provided relatively long-term WE. The AFDC Homemaker Demonstrations trained eco-nomically disadvantaged women to provide in-home care to the disabled and the elderly (Bell and Reesman, 1987). Participation rates in these relatively expensive treatments were high and similar services were generally unavailable to the controls (Masters and Maynard, 1981, p. 148; Bell and Reesman, 1987, p. 14). Therefore, in these two studies the experimental impacts can reasonably be interpreted as approximating the impact of the “treatment on treated.”

As suggested by the number of studies surveyed in Table 10.4, there have been fewer experimental evaluations of the impacts of employment and training programs for adult men and especially for youths. As a result, the evidence based on social experiments is more fragmented. Nevertheless, the evidence suggests that programs that o¤er training can raise the earnings of economically disadvantaged adult males, but programs that focus on JSA or WE appear to be ine¤ective or sometimes worse. Earnings impacts of the San Diego CWEP program, the Baltimore Options program, and the NSW Demonstration were small or negative for disadvantaged adult men. By contrast, the impacts reported in programs that o¤ered training opportunities, San Diego-SWIM program, GAIN, and the NJS, were larger and statistically signi…cant. In particular, the NJS found that economically disadvantaged adult men experienced earnings gains similar to those achieved by adult women (Orr, et al., 1994. p. 82).

The evidence from experimental evaluations for youths is not encouraging. As shown by the last panel of Table 10.4, the results suggest that the array of services currently o¤ered do little to raise youth employment and earnings. For example, the prolonged WE provided to disadvantaged high school dropouts in the NSW Demonstration had no e¤ect on their earnings during the eight years after the treatment was o¤ered (Couch, 1992). Similarly, the JOBSTART demonstration, which provided disadvantaged youths with services similar to those o¤ered by the comprehensive Job Corps program, but without the residential living centers, did not generate signi…cantly higher earnings for the treatments during the four year follow-up period (Cave, et al., 1993). Finally, the NJS …nds no evidence that youth served by JTPA bene…t from its relatively low cost training services. In fact the short-term point estimates for the males were actually negative.

Another …nding highlighted in Table 10.4 is the correspondence between earnings im-pacts and employment imim-pacts. In most cases large earnings imim-pacts are accompanied by signi…cant impacts on employment rates. Moreover, in most of these studies analysts measure employment rates at the quarterly level and information on hours of work are un-available. When such measures are available, hours impacts also can be a signi…cant source of earnings gains. (See, e.g., the NSW Demonstration, Hollister, et al., 1984.) Indeed, there are only two cases in the table for which the long-run earnings impacts are signi…cant, but

not the impact on employment rates. This evidence underscores the concern that because access the government employment and training programs raises earnings through higher employment rates, displacement of non-participants may mitigate the net social bene…ts reported for these treatments in conventional cost-bene…t analyses.

The experimental impacts reported in Table 10.4 indicate that the impact of the op-portunity to participate in particular employment and training services varies substantially among demographic groups. The WE services provided in the NSW demonstration were e¤ective for adult women, but not youths; the WE provided in the San Diego CWEP demonstration was more e¤ective for female welfare applicants than for their male counter-parts. The JSA and training experiences provided in the San Diego SWIM demonstration also had a larger impact on women than on men.⁹⁷ Finally, the NJS reported striking di¤erences between the impact of JTPA services on adults and youths. These results raise the issue of the importance of impact heterogeneity in this literature.

Just as this impact heterogeneity is found among di¤erent demographic groups, it also is often found among di¤erent sites in the same study. When experimental impact estimates for the same program are available for di¤erent sites, it is common to …nd that the impacts vary among sites. For example, as shown by Table 10.4, the results from the GAIN program and Minority Female Single Parent Demonstration (MFSP) reveal substantial variation in impacts among sites. Similar variation in experimental impacts also is reported among the 10 sites in the NSW Demonstration and the 16 sites in the NJS. (See Maynard, 1980, p.

83, Masters and Maynard, 1981, p. 85; and Heckman and Smith, 1998c.) At the very least, this evidence of heterogeneity in impacts among sites raises the question of the external validity of these evaluations, i.e., whether their results can be extended to other settings.

For policy purposes it is important to know whether the di¤erences in site impacts arise from di¤erences in the skills of program operators and trainers, program organization, or the characteristics of those who are served.

The experimental evidence can shed some light on how heterogenous the impacts are among those served by these programs. An important question in this regard is whether government training programs generate di¤erent returns for participants depending on their observed and (to the econometrican) unobserved skills. If returns are smaller for the least skilled, then policy makers would be faced with the di¢cult question of whether to reallocate expenditures toward less “needy” participants. In 1981, U.S. policy makers in fact made the opposite decision when they directed that employment and training expenditures be

97These di¤erences in experimental impacts are not the result of di¤ering participation rates in the programs by women and men. In the San Diego SWIM Demonstration participation rates in programs services among female (i.e., AFDC-FG) and male (AFDC-U) treatments were nearly the same. Male controls were less likely than female controls to obtain the same services elsewhere. See Freidlander and Hamilton (1993), p. 22, Table 3.1.

targeted to a more economically disadvantaged population (Barnow, 1987). An important policy question is whether this decision improved or worsened the returns from these social programs.

Neither the experimental nor the non-experimental evidence provides a clear answer to the question of whether the impacts of these programs vary with participants’ skills.

But the experimental evidence does suggest that the least able participants among the low-skilled populations served by these programs bene…t the least from them, especially when the programs provide CT and OJT opportunities. To illustrate these points, Table 10.5 presents the experimental impacts by the prior skills of participants for several social experiments. The measures of skill di¤er among studies, but as indicated by the controls’

earnings during the follow-up period, these di¤ering measures of skill correctly identify individuals likely to perform poorly in the labor market. In the GAIN and NJS studies more skilled persons bene…ted more from access to the program’s services than did less skilled persons. However, as the table demonstrates, in some programs, such as the NSW and the San Diego CWEP Demonstrations, the least skilled experienced larger gains. Signi…cantly, these programs provided treatments with WE. As explained in Section 2, the purpose of this service is to provide a job experience to individuals with poor employment histories so that they can develop acceptable “work habits.” By design, therefore, it might be expected that this service would provide greater bene…t to less skilled participants than to more skilled participants who already possess such skills.