QWL Applications in Public Agencies: Do Sucess Rates Reflect a Positive-Findings Bias?

全文

(1)IHT'L. J. OF PUB. ADMIN., 15(6). 1263-1279 (1992). QWL APPLXCATXOHS XH PUBLXC AGEHCXES:. DO SUCCESS RATES REFLECT A POSXTXVE-FXNDXHGS BXAS?*. Robert T. Golembiewski. Department of Political Science. University of Georgia. Athens, Georgia 30602. Ben-chu Sun. Graduate Program of Public Administration. National Chengchi University. Mucha, Taipei. Taiwan, R.a.C.. AJ:)stract. This article builds on reports of substantial success rates in Quality of Working Life (QWL) applications, and tests the criticism that these attractive outcomes merely reflect wimpish methodology. As the rigor of research designs increases, critical observers predict, so also will outcomes deteriorate. This panel of QWL applications, N = 44, does not support a positive findings bias in public-sector applications, where QWL success rates approximate those in business applications. Quality of Working Life (QWL) proponents make strong cases that management bottom-line needs can be met, while empowering employees and approaching representative political ideals in organizations, in both business and public sectors. For example, two. 1263 Copyright © 1992 by Marcel Dekker, Inc..

(2) 1264. GOLEMBIEWSKI AND SUN. QWL theorists (1) propose that four dimensions underlaying the humanization of work can and should be attended to: Security: the worker's freedom from fear o and anxiety concerning his health, safety, income and future employment. Equity: the worker receives compensation o commensurate to his contribution to the value of a service or product. Individualism: work should stimulate the o development of the individual's unique abilities, craftsmanship, and the capac~ty for continued learning rather than force him into a mechanized role. o Democracy: • the worker [is not] a passive object • Wherever feasible, workers should manage themselves; authoritarian, hierarchical control should be replaced by cooperative self-managed groups. Especially in the public-sector, however, some observers dismiss such claims for both employees and employing organization. Some see the claims as illusions, and often cruel ones. Such observers propose that public-sector features are refractory to QWL values and designs, and that success rates are low. Brief attention below will be devoted to QWL as illusion, since other work provides details (2). Basic attention here goes to QWL as artifactual, with specific reference to the criticism that QWL evaluative studies lack methodological and design rigor. Consequently, despite SUbstantial success rates reported in over three dozen published surveys. QWL API. of aPI positj artifa rigor propos and po buttre editor result T. panel artifa presen assemb. T. cases, introd short detail QWL ap detail genera F. occur, the fel dissem. S. major ~ applicc a rangE.

(3) AND SUN. lould fear lfety,. ltion :he the. :e. ,acity rce a Ie,. tlould ad. :lome :; and. )ry to. Ire. !WL as. :tual,. 'L. n. s. veys. QWL APPLICATIONS IN PUBLIC AGENCIES. 1265. of applications (3), some critics urge that a positive-findings bias operates to induce artifactually-high estimates of outcomes. As the rigor of evaluative studies increases, this criticism proposes, so also do outcomes become less attractive and positive. At times, this alleged bias is seen as buttressed by powerful reinforcers--e.g., that editors and their boards prefer to publish positive results. This paper is part of ~ series testing a large panel of QWL applications for illusion and artifactuality (4). The panel is the largest presently available (N = 231), and details about its assemblage and character are available elsewhere (5). FEATURES OF PUBLIC-SECTOR APPLICATION The QWL sub-sample of interest here contains 44 cases, and its major features need early introduction. No completely-satisfactory way exists, short of reading each case in the panel of 231, to detail the characteristics of the 44 public-sector QWL applications. But Exhibit 1 presents sUbstantial detail and four points highlight useful generalizations about .the summary. First, about two-thirds of the applications occur at sub-federal levels of government. In short, the federal government does not take the lead in disseminating QWL. second, QWL applications have been made in the major government areas or functions. civil applications clearly dominate, but the panel includes a range of others--military, schools, and so on..

(4) 1266. GOLEMBIEWSKI AND SUN EXHIBI'l' 1. Features of PUblic-sector OWL Applications. I.. Locus 1. Federal 2. sub-Federal, usually Local Government 3. Combined II. Function Involved 1. civil 2. Military 3. Police 4. Schools 5. Combined III. Primary Level of Application 1. Operators 2. First-line supervisors and subordinates 3. supervisors only 4 Executives and mid-managers 5. Multiple IV. Primary Designs 1. Alternative work schedules 2. Job enrichment 3. Feedback, usually using surveys 4. Union/management collaboration 5. Team building 6. Incentive plans 7. Management by objectives 8. Combined . =. N. 10 2. N = 33. 4 1. N. =. 14 5 1. 2 21. N = 9. 7 5 2 3 2 2. 12. Third, OWL applica~ons tend to emphasize operating and supervisory levels, but Exhibit 1 shows that about half of the cases encompass multiple levels of organization. Specifically, this refers to the inclusion of three or more levels, typicallY including operators and their first-line supervisors. Fourth, the 44 public-sector OWL applications include a broad representation of designs. About 30 percent of the applications get classified as Combined, which implies the useful intent of reinforcing and supplementing effects--as when. changes employe that hi A. 32. 2 4. QWL APPL:. designs provide review to chan or proc Of schedul often r program employe percent increas policy. design, arrange in .skil various the sin, generat. greater Br, 44 OWL. the pre! that ex' public-l multipl!. interver grasp--c not an c.

(5) AND SUN. N. 10. =. 32 2 N. 33 4 1 2 4 N. 14. =. =. 5 1 2. 21 N. =. 9 7 5 2 3 2 2. 12. shows ~rs. to. .sors.. ms. Lt 30. QWL APPLICATIONS IN PUBLIC AGENCIES. 1267. changes in policies, procedures, or structures are employed to reinforce interaction-centered efforts that highlight worksite inadequacies. A brief review of the cases in which single designs dominate serves two purposes. It not only provides information about frequencies, but that review also highlights the three major OWL approaches to change--via interaction, structure, and policies or procedures. Of the single designs, alternative work' schedules--especially flexible workhours--are most often relied on. As is widely known (6), such programs generate attractive outcomes for both employees and their organizations in a very high percentage of cases. This intervention seeks to increase responsible employee freedom via a change in policy. Job enrichment is the second most popular design, and it seeks the same end via structural arrangements that permit the employee greater scope- in skills, self-discipline, pacing, and so on. Various feedback designs rank third in usage among the single designs. They emphasize interaction that generates valid and reliable information that permits greater responsible freedom at work. Broadly, this review of some major features of 44 OWL applications in the public sector implies that the present panel has quite a comprehensive reach that extends to different loci, includes major pUblic-sector areas or functions, encompasses multiple levels, and involves the full range of OWL interventions. This OWL reach implies that its grasp--as reflected in sUbstantial success rates--is not an artifact of selectivity or narrow biases..

(6) 1268. GOLEMBIEWSKI AND SUN TABLE 1. QWL Success Rates on Hard-Criteria outcomes,. public Versus Business sectors. QWL outcomes, in percent 1. 2. 3.. 4.. -------------------------------------Definite. N. =. Balance of Highly positive positive No and and Intended Intended Appreciable Negative Effects Effects Effects Effects. -------------------------------------------. Private Sector Public Sector. 185. 71.4. 20.5. 4.9. 3.2. 44. 59.1. 27.3. 9.1. 4.6. QWL APP. substa of the balanc additi bit, b applic. M. availa statem First, from t studen abilit the ra presen S. PUBLIC-SECTOR OWL AS ILLUSION. Earlier papers (7), focus on success rates--that is, on whether QWL achieves its intended effects. Table 1 summarizes the data from a comparison of the public and private sectors. Note two points. Thus, success rate here is defined in terms of hard criteria\outcomes--changes in objectively-assessed productivity, cost data, turnover, and so on. A similar table for "soft" measures--self-reported changes in attitudes, and the like--provides essentially the same results. Moreover, two cases could not be assigned to either of the two sectors. Hence N = 229. Table 1 presents no great interpretative challenge and, on its face, supports two conclusions. Public-sector outcomes of QWL applications generate. outcom pre- v assign variab direct costs "Large level assign expect all ca a "4" T. conven disagr Failur assign.

(7) AND SUN. ., 4.. gative ffects. 3.2 4.6. --that. ... the 'hus, ed. es rs.. ions. ite. QWL APPLICATIONS IN PUBLIC AGENCIES. 1269. substantial success rates. In sum, over 85 percent of the cases are rated as having at least a definite balance of positive and intended effects. In addition, the outcomes in the two sectors differ a bit, but only a moderate advantage goes to business applications. Measurement details underlying Table 1 are available elsewhere (~), but several summary statements provide useful context for Table 1._ First, all codings in this and related reports derive from three independent raters, all advanced doctoral students in the behavioral sciences. Inter-observer abilities vary a bit for different codings, but in the range of the low .90s. So coding unreliability presents no major problems. Second, strict rules guide assessments of QWL outcomes, with the typical evaluative study reporting pre- versus post-scor~::; on several variables. A "1" assignment in Table 1 requires that changes in most variables reported in a study are in the expected direction--toward higher productivity and lower costs--and are large in at least half the cases. "Large" here means a change that achieves the .05 level or is greater than 10 percent. A "2" assignment also requires that most changes are in the expected direction, but fewer are large. Virtually all cases containing large and unexpected changes get a "4" code. Third, in general, conservative scoring conventions prevail. For example, if observers disagree about an outcome code, discussion follows. Failure to achieve agreement results in the assignment of the lowest outcome score-in-contention..

(8) 1270. GOLEMBIEWSKI AND SUN. QWL APP. Note that inter-observer reliabilities are calculated before discussion.. a 0-9 have t or 1 s. PUBLIC-SECTOR OWL AS ARTIFACT. The outcomes in Table 1 are attractive, and even formidable. But do they reflect less about relationships in nature than they do about a positive-findings bias (PFB)? Testing for PFB rests on three methodological emphases. Two measures of methodological rigor are introduced; conventions about measuring the outcomes of QWL applications are reviewed; and the properties of statistical methods are sketched. 1.. Two Measures of Rigor. The summary of QWL outcomes in Table 1 tells us nothing about the methodological quality of the 231 evaluative studies, and so the apparent success rates there may reflect the inadequacies of research design. This possibility is taken very seriously here, and three estimates are made of the rigor of each evaluative study. Two separate estimates deal with the rigor of methodology and design (MjD), while the third estimate deals with the degree to which each study successfully deals with threats to internal and external validity (lEV). The correlation of the two MjD measures is high--r=.91. So the assessments of rigor admitted to analysis here include one MjD and one lEV measure. Woodman and wayne's MID score Based on criticisms (9) of Terpstra's (10) original formulation, Woodman and Wayne (11) propose. 1. 2 3. 4. 5. 6. 7.

(9) I AND SUN. QWL APPLICATIONS IN PUBLIC AGENCIES. :::ulated. a 0-9 measure of MID rigor. Their nine dimensions have been slightly modified by Sun (12), and assign 0 or 1 scores for: 1. Sampling strategy: 1 indicates a full census or a representative sampling strategy like random sampling, stratified sampling, or cluster sampling. A nonrepresentative sampling strategy or an unspecified sampling plan is marked O. 2. Sample size: 1 is assigned when N > 30, 0 when N .:5. 30. 3. Control or comparison groups: 1 indicates the presence of such a group, and 0 in~icates its absence. 4. Random assignment utilization: 1 represents use of random assignment, and 0 its absence. 5. Measurement strategy: 1 indicates longitudinal measurement, or measures taken at two or more times, and 0 indicates a cross-sectional or one-time measurement. 6. Reliability and validity of measures employed: 1 represents reliability of measures 2 .6, along with some evidence of validity; 0 stands for no reliabilities reported, or reliabilities < .6 with no validity evidence. 1 7. criteria for dependent variable(s): reflects the use of some objective criteria; 0 stands for perceptual data only.. ld even. .cal : are :comes !rties. Is us 231. rates. ere,. ch. fNith e the ich il and :! two of and. 'pose. 1271.

(10) 1272 . GOLEMBIEWSKI AND SUN 8. . significance level:. 1 indicates a. probability of type I error. ~. othe. .05; 0. indicates p > .05. 9. . 'QWL }. each. Use of a statistical analysis procedure:. 1. poor. Following Morrison (13), an estimate of internal. asse. is yes, and 0 is no. Morrison's I/E Score. and external validity provides additional perspective. and. on the rigor of QWL studies.. both. Her approach rests on. Campbell and stanley (14), who identify four threats. smal. to external validity and eight threats to internal. and. validity.. nega'. In their usage, "external validity" refers. to the generalizability of results to other settings,. asso~. and "internal validity" refers to the likelihood that. cate(. the relationship between an independent and dependent. meaSl. variable actually exists. Morrison's liE score varies from 0 to 12, depending on whether or not each of the described below is reduced.. t~reats. Details are available. elsewhere (15), but note here that the eight threats. teste. In ad. to internal validity involve eliminating or reducing. non-r. these possible effects:. analy. non-treatment factors; aging. or development; sensitivity to testing instruments; measurement. insta~ility. in the instruments;. both diffe. regression to the true mean; non-random selection;. as. experimental mortality over the period of. this 1. mo~. observation; and interaction of selection and maturation.. In addition, Morrison's liE score. assesses the design's management of four threats to external validity: any reactive or interacting. findil Thus, show. effects of testing; interaction of selection effects. Moreo'. and the experimental variables; generalizability to. comhil.

(11) UN. QWL APPLICATIONS IN PUBLIC AGENCIES. L. other settings; and multi-treatment interference. A res~arch design that reduces or eliminates each of the threats to validity is scored a 12. The poorest design is rated a O.. ~. 1273. 2.. Alternative Measures of Outcomes. Each evaluative study is rated for two assessments of outcomes: a hard-criteria assessment; and a global assessment, which takes into account both "hard ll and "soft" data. Moreover, due to the small number of applications with outcomes rated "3" and "4"--that is, as having no appreciable effect or negative effects--tests of rigor and outcome associations are run for both three- and four category measures of outcomes. The three-category measure aggregates "3" and "4" ratings. 3.. Statistical Methods. All associations of rigor and outcomes will be tested by one-way analysis of variance, basically. In addition, 'any cases are shown by ANOVA to include non-random variance (P 5 .05) will be subjected to an analysis of all possible paired-comparison to assess both the magnitude as well as the direction of all differences. The Least Significant Difference test, as modified for unequal sub-sample sizes, is used for this purpose. In sum, perfect support for the positive findings bias requires rather stringent conditions. Thus each rigor and outcome association would have to show statistically-significant variance by ANOVA. Moreover, every possible pair of rigor and outcome combinations would be in a direction consistent with.

(12) 1274. GOLEMBIEWSKI AND SUN. QW. TABLE 2. Illustrative ANOVA, OWL outcomes and I/B Validity,. Bard-Criteria Assessment, Public-Sector Applications. of Ho co: on in. lIE Validity Scores. -------------------------N. Outcomes 1. Highly positive and intended effects 2.. 3.. 4.. Definite balance of positive and intended effects No appreciable effects Negative effects. Standard Mean Deviation ANOVA. vel. --------------------------. arE. 26. dit. gre 6.0. 3.52 F=3.21. 12. 4.7. 2.84 F= .033. 4 2. 9.8 2.5. 2.50 0.71. the positive-findings bias. specifically, this means that as rigor increases, the outcomes deteriorate. Finally, perfect support of the positive-findings bias requires that each paired-comparison is statistically significant as well as in the appropriate direction.. Testing for positive-Pindings Bias. This analytic procedure is not complicated, as Table 2 illustrates for one of the 8 tests of the PFB hypothesis. There, ANOVA indicates that non-random variation does exist in the association between Morrison's measure of internal/external validity and QWL outcomes. In addition, a SUbstantial proportion. att att. midI bia!. encc well well even publ SUIDm2 iHus. signi outcc the c test equal contr. findi:. balan.

(13) ID SUN. ~y,. .ODS. A. 21. 033. ~ans. lS. PFB )m. md. Lon. QWL APPLICATIONS IN PUBLIC AGENCIES. 1275. of the variance in scores is explained (eta2 = .19). However, the direction of the associations is not consistent with the positive-findings bias. In fact, only two of the six possible paired-comparisons fall in the appropriate direction--(l) versus (3), and (2) versus (3). only in one-third of the cases, then, are QWL outcomes less attractive when rigor is greater. Moreover, although both paired-comparisons directionally consistent with the PBF hypothesis do attain P ~ .05, one of the four contrary cases also attains statistical significance. In sum, the case illustrated in Table 2 provides middling and mixed support for the positive-findings bias hypothesis. The eight total tests of the PFB hypothesis- encompassing two measure of rigor, hard-criteria as well as global assessments of outcomes, and three- as well as four-category estimates of outcomes--provide even less support for a positive-findings bias in public-sector QWL applications. See Table 3 for summary details. Paramountly, only the ANOVA run illustrated in Table 2 achieves statistical significance, so seven of eight distributions of outcome and rigor are random only. Note also that the directions of paired-comparisons--the most gentle test of the positive-findings bias--are in relatively equal proportions consistent with the hypothesis, and contrary to it.. DiSCUSSiON. In sum, this paper shows that a positive findings bias contributes very little to the definite balance of attractive QWL outcomes in the public.

(14) 1276. GOLEMBIEWSKI AND SUN. Q\ol. TABLE 3. Summary Table, Tests of positive-Findings Bias,. PUblic Sector Applications. Raw Data 1.. 2.. 3.. 4.. 5.. OVerall ANOVAs attain P .s .05 Paired-comparisons in direction consistent with hypothesis and statistically signifi cant by LSD test, Modified Paired-comparisons in direction consistent with hypothesis Paired-comparisons in direction contrary to hypothesis Paired-comparisons in direction contrary to hypothesis and statis tically significant by LSD test, Modified. Percent. 1 of 8 cases. 12.5. 2 of 36 cases. 5.6. 21 of 36 cases. 58.3. 15 of 36 cases. 41.7. Th is. la: la; 1 of 36 cases. 2.8. pa1. prE. pOE. sector.. QWL outcomes do not deteriorate with. the. heightening rigor. This finding encourages QWL applications in the. twc. public sector, and also gets reinforcement from two. dis. other sources. ThUS, public-sector applications seem to have virtually the same pattern of success rates. h}'I:. as do business applications.. See Table 1.. Moreover,. pes. des. the present public-sector panel encompasses four. (4). kinds of QWL interventions that are widely. res. applicable.. eff. These four types of QWL interventions. sel arg. include:. o. Human Processual, which seeks to improve personal and group functioning, as by team building. out.

(15) UN. QWL APPLICATIONS IN PUBLIC AGENCIES. o. 1277. Sociotechnical Systems, which design micro organizations to integrate technological and social systems to create flexible structures and high personal involvement, in autonomous teams. o Technostructural, which can apply at macro as well as micro-levels--e.g., workflow reorganization and matrix models as well as job enrichment. o Combined, which rely on interventions from two or more of the above classes. Real tethers exist on present findings, however. Thus, the population of public-sector interventions is not large (N = 44), although it represents the largest available panel. Subsequent studies of larger populations may not replicate the present pattern of results. In addition, studies in the present panel typically are pre-test versus short post-test designs, and longitudinal studies may find that effects decay quickly. This research also cannot be definitive about two other major issues, but these do nothing to discourage QWL interventions. First, the basic hypothesis here may be misguided. That is, the positive-findings bias assumes that rigorous research designs will tend to isolate outcomes rated (3) and (4)--no appreciable effects and negative effects, respectively. In contrast, it may be that negative effects will be quite obvious to all but the most self-interested observers. If so, research rigor is arguably not crucial, except perhaps to differentiate outcomes (l) from (2)..

(16) 1278. GOLEMBIEWSKI AND SUN. QWl. Second, a kind of dual and simultaneous incompetence may exist. An incompetent plan for application often may be paired with an inelegant design for research. This possibility sets aside the positive-findings bias. Here, the problem is not in QWL, but is relocated to the intervenor/researcher. NOTES. •. An earlier version of this paper was presented. at the Annual Meeting, American Political Science. Association, September 3, 1989.. REFERENCES. (1) Herrick, Neal, Q., and Maccoby, Michael. "Humanizing Work," in Davis, L.E. and Cherns, A.B., (editors), The Ouality of Working Life, The Free Press, New York, 1975, pp. 63-77. (2) Sun, Ben-chu. Quality of Working Life Programs. unpublished doctoral dissertation, University of Georgia, Athens, Georgia, 1988; Golembiewski, Robert T., and Sun, Ben-chu. "Enriching Work and Empowering Employees." Paper presented at Annual Meeting, American Society for Public Administration, Miami, Florida, April 1989; Golembiewski, Robert T., and Sun, Ben-chu. "QWL Improves Worksite Quality." Human Resources Development Ouarterly ~ {September 1989): in preSSj Golembiewski, Robert T., and Sun, Ben-chu. "Positive-findings bias in QWL research: A Comparison of Public and Business Sectors." Public Productivity Review (1990): in press.. (3) Sun, Ben-chu. Programs, 1988.. Quality of Working Life. (4) Golembiewski, Robert T., and Sun, Ben-chu. "Enriching Work and Empowering Employees,1I 1989; Golembiewski, Robert T., and Sun, Ben-chu. "QWL Improves Worksite Quality," September 1989.. (5. 19 (6. Jr Wo (7 liE. r. 19 llQ. (8. 19 (9. Fi In An Te. (1. In' of of (1. 19 (1. Or' (1. Ex Re 19 (1. 19.

(17) JM. QWL APPLICATIONS IN PUBLIC AGENCIES. (5) Sun, Ben-chu. 1988.. =. 1279. Quality of Working Life Programs,. (6) Golembiewski, Robert T., and Proehl, Carl W., Jr. "Public-sector Applications of Flexible Workhours." PUblic Administration Review 40: 72-85. (7) Golembiewski, Robert T., and Sun, Ben-chu. "Enriching Work and Empowering Employees," April 1989; Golembiewski, Robert T., and Sun, Ben-chu. "QWL Improves Worksite Quality," September 1989. (8) Sun, Ben~chu. Quality of Working Life Programs, 1988. (9) Bullock, R.J., and Svyantek, D.J. "positive Findings Bias in Positive-Findings Bias Research." In Chung, K.H., (editor), Proceedings: 221-224. Annual Meeting of the Academy of Management, Dallas, Texas, 1983. (10) Terpstra, David E. "Relationship Between Methodological Rigor and Reported Outcomes in Organization Development Evaluation Research." Journal of Applied Psychology ~ (December 1981): 541-543, (11) Woodman, Richard W., and Wayne, Sandy J. "An Investigation of Positive-Findings Bias in Evaluation of Organization Development Interventions." Academy of Management Journal 28 (December 1985): 889-913. (12) Sun, Ben-chu. 1988.. ouality of Working Life Programs,. (13) Morrison, Peggy. "Evaluation in OD." Group & organization Studies ~ (March 1978): 42-70. (14) Campbell, Donald, T., and Stanley, J.C. Experimental and Ouasi-Experimental Designs for Research. Houghton Mifflin, Boston, Massachusetts, 1963. (15) Sun, Ben-chu. 1988.. Quality of Working Life Programs,.

(18)