REFLECTIONS ON A HALF CENTURY IN
PROGRAM EVALUATION
James W. Altschuld
ABSTRACT
A retrospective trip through 50 years from graduate school to work in evaluation is the foundation of this article. It affords a way of looking at how the training of evaluators and what they do in conducting evaluations have changed, how personal experiences shaped perceptions about a field, and developments that transformed evaluation into the field that it is now. The author’s graduate program is examined for its strengths, the guidance it provided him with, weaknesses, and what was missing and needed. His long-term involvement in many projects reflects both the evolution of evaluation practice and knowledge growth. A historical view is valuable for understanding the progress that has been made and, at the same time and surprisingly, shows that issues encountered many years ago are in ways and to a degree still pertinent and remain with us. This is especially true in regard to the teaching of a new generation that will carry evaluation forward. The past is indeed a backdrop for the future and the seeds of current concerns were evident a long time ago. Conclusions drawn from the five decades lead to what we might see as evaluation moves fully into the 21st century. The ultimate goal is to have evaluators individually and as a collective always question who we are and the premises that underlie our actions.
Keywords: evaluation history, evaluation training, training issues, trends
James W. Altschuld, Professor Emeritus, Member of the Inaugural Class of the University’s Emeritus Academy, The Ohio State University, Columbus, U.S.A.
E-mail: [email protected]
Manuscript received: May 12, 2016; Modified: August 11, 2016; Accepted: September 8, 2016
Introduction
The opportunity to look back on a professional career that began in 1965 with graduate study is unique and provides a perspective of what evaluation was and could be. A few words about me before beginning! I retired in 2004 after 39 years of teaching and doing research on evaluation and needs assessment at The Ohio State University (OSU). I stayed there until 2010 via projects, consulting, writing, and presentations. Related efforts have been ongoing since but at a reduced pace. More about what I did comes later.
Given my tenure in evaluation, what has emerged from a nebulous entity into a ‘near’ discipline or profession and what new forms and attributes of practice have evolved? For answers, I review my doctoral experiences (Table 1), career development, perceptions of how well graduate education prepared me for what would come later, and what the future of evaluation might hold. Some conclusions are then offered. While much has changed and improved, a set of problems observed a long time ago are still with us (see Tables 2 and 3, particularly in relation to Table 1). Occasionally and where pertinent, comments about the present are in the historical discussion which is organized around 3 questions.
Question 1: What is and can be learned from my preparation and experience?
Question 2: What was missing in my preparation for evaluation? Question 3: How has the field changed over time?
Question 1: What Is and Can Be Learned From My
Preparation and Experience?
Academic Preparation
I entered the doctoral program in 1965 in educational research and development at the Ohio State University (OSU) after a master’s in chemistry and work as a chemist. To be honest, my heart was never fully into chemistry. In 1965, I embarked on the new graduate program as the U. S. Elementary and Secondary Education Act with its evaluation mandates had just been enacted and training for evaluators was almost non-existent, evaluation was in its infancy.
My first advisor was Egon Guba who gained fame for qualitative evaluation methods and then Daniel Stufflebeam chaired my committee until completion of the doctoral candidacy exams. He was formulating his well-known Context, Input, Process, and Product (CIPP) model (Stufflebeam,
1973) and I remember an early presentation of it in a class about 1966. These mentors suggested that I was a good fit for evaluation and an exciting venture began.
I had gone into a setting that was quite foreign to me. My academic program, Table 1, consisted of educational studies to acquaint a student with what education entailed (about 35% of the coursework), methodology and measurement, and statistics (more than 40%), and educational research and evaluation for the rest. This part of my graduate studies required over two years to complete.
The doctorate was awarded in 1970 and the program was reasonable, but it was not a fully integrated graduate plan and some important things were missing. As for good aspects, I was involved in an evaluation center under Stufflebeam and after that the Columbus Ohio Public Schools, the State of Delaware Department of Public Instruction and OSU’s Center for Vocational Education until becoming an assistant professor at the university in 1978. The learning over these eight years included:
Table 1
Doctoral Program in 1970s
Area Types of Courses Commentary
Educational Studies
7-9 courses in the history and philosophy of education, curriculum structure, learning theory, guidance and counseling, change theory, etc.
An overview of how schools function and the design of educational programs.
Methodology
Sociological methods with related courses in psychology.
Content was statistics (correlation, analysis of variance), measurement, instrument development, item writing.
Exposure to a cross section of methodology with writing and designing items (appropriate for a person in educational research).
While not an in-depth specialization, coupled with the other parts of the program, it was a solid base for what I would do in the future.
Educational Research, Development and Evaluation
Varied content about research in schools and educational systems, a course on evaluation (there was only one), seminars on related topics and readings, independent studies.
Complemented with several classes in psychology and a student led review group.
Getting to know a cohort who were undergoing similar preparation, establishing friendships/contacts that have lasted through the years.
This facet of academic work would prove to be invaluable in my work life.
Rounded out with involvement in projects.
- How programs and projects are funded; - Proposal writing and the competition for funds; - The design of innovations;
- Practical ways in which innovations might be evaluated;
- Negative views of evaluation held by many non-evaluators and administrators as well as why they have these perceptions;
- The politics of evaluation and the need to be tactful in addressing sensitive concerns and issues (an art which I have not always successfully practiced nor mastered);
- Thinking like a developer of new initiatives and an evaluator almost simultaneously to the extent possible.
Could such understandings have been gained in university classrooms? Yes, although some probably could not. In this regard former students who were in my OSU evaluation classes commented over time that more emphasis should have been on the politics of evaluation. They had encountered situations for which they did not feel adequately prepared. I generally agree.
However, until one is in a difficult predicament it is hard to sense and convey what evaluators run into and the pressures that can be exerted on them. Let me illustrate through a recent project in which a local site repeatedly attacked a single finding out of 15 to discredit the evaluation and me, the evaluator. This was communicated to the national agency officer without the evaluator being present. Evidently the other results were fine and used extensively by site staff yet the attack was unusually intense about the one outcome. Explaining this is like telling someone not to touch a hot stove when they are not in the kitchen and later they encounter the problem and possibly touch it anyways. Shortly after the attack occurred I left the project and it lost government support.
As for what was missing in graduate training, deficiencies were apparent. Quantitative methods were prominent in the sociology, psychology, and educational research courses in the 1960s and 1970s and are dominant again. In the late 1980s and 1990s a shift toward qualitative approaches seemed to take place but that has now almost completely dissipated across the world with the earlier pattern reasserting itself in training programs (LaVelle, 2014). Qualitative methodology was not there previously and today is absent at the master’s and doctoral levels. Just like when I started in the early formative days of evaluation as a field, quantitative prevails.
From the entries in Table 1 another discrepancy is subtle but notable. Where are the evaluation courses? How did one learn about evaluation? Where was the history (even though it was a nascent one) of the field, what models and theories of evaluation should an aspiring evaluator know, what about research on evaluation and strategies for designing and implementing evaluative studies. What about needs assessment, cost-benefit analyses, and mixed methods? Was even referring to a field of evaluation in the 1960s appropriate? Textbooks for those entering evaluation were not available? In my opinion, the first major one was by Worthen (an OSU classmate) and Sanders (Worthen & Sanders, 1973), three years after my degree. Now there are newer editions of it, Fitzpatrick (Fitzpatrick, Sanders, & Worthen, 2011) became the first co- author, and other texts have appeared.
This was the status as the 1970s dawned and an indicator of the limited scope of things. Possibly like others with my background we were thought of as generalists who could provide a wide range of service. My perception is that for most of my cohort, evaluation understandings came through involvement and participation than by other means. That was okay but more was needed.
Career Life Into Semi-retirement
I began as an evaluator in 1968 at the evaluation center two years before receiving the doctorate. Subsequent employment as noted was in a big city school system, a state department of education, and a national vocational and technical education center. From 1968 to 1978, I was never a full time evaluator and had responsibilities from program planning and evaluation and project management to professional development for educators and administrators. I presented on accountability systems, writing proposals for funding, and the use of outcome and process objectives.
In one unusual twist, my job was switched to that of an auditor of educational programs and in the oddest circumstance I was required to audit a project that I designed. (Under my U.S. Constitutional 5th amendment rights against self-incrimination I refuse to reveal the results of the audit report.) Possibly like others with my background we were thought of as generalists who could provide a wide range of service. Given my success, that view was accurate.
Learning was mostly on-the-job, from the literature, and through workshops on qualitative methodologies, cost and benefit analysis,
epidemiology, and other techniques. The latter way to grow and improve was a constant during my years of professional practice. One other event from the first 10 years must be underscored. A co-worker (J. Lave-Gudaitis) had been doing an assessment of needs and shared instruments and documents that she located. That’s how I became familiar with the ideas of B. R. Witkin who later became a dear collaborator. In 1978, I became an assistant professor in educational research and evaluation at OSU. There, I revisited needs assessment (NA) and created a course sequence about it - almost the only one of its kind in education in the U.S. NA remained a career focus with 8 books, a New Directions Issue, and research articles being published. The impact of my two friends cannot be emphasized enough and I am ever grateful to them.
At the university originally I was to teach basic research and statistics, and an introduction to educational evaluation along with resurrecting an evaluation emphasis in the College of Education (only scattered remnants of what long gone actors had produced could be found). This was a challenge. It was a long time since I had been in formal academia. I did not have many relevant teaching materials on hand but I had a fairly extensive base of experience. Creating, refining, and delivering courses represented a steep learning and work curve but one that I enjoyed.
Parenthetically, there was some antipathy toward evaluation from a few of my immediate faculty unit members who adhered to a narrow and purist view of methodology. This was a negative factor in re-establishing evaluation in the College and even now the preparation of professional evaluators in it could be described as moribund.
By 1995 I was a full professor, teaching evaluation classes (including needs assessment) and some on research methods. From the early 1990s I also did project work in 3 OSU centers up to 2010 with related efforts going on up to the present. Currently I am semi-retired. So history tells about where I came from but what can we learn from it (Question 2)?
Question 2: What Was Missing in My Preparation for
Evaluation?
In Table 2, are strengths from my background that I think have relevance for what we see today. Table 3 contains weaknesses and gaps that I felt and what might be needed for evaluation as it becomes more mature. When looking at the two tables an important thing to note is that issues and problems
seen long ago persist to varying degrees up to this point in the 21st century. We still have not achieved agreement as to what should be the training of evaluators.
Table 2
Strengths From Prior Training and a Career in Evaluation
Area Strengths Comments/Discussion
Program planning and
implementation
Hands-on learning in designing and implementing programs from inception to completion.
Understanding outcome and process objectives.
Seeing where weaknesses in programs might be, gaining insights on where to focus evaluation processes.
Becoming intimate with what an innovation is trying to do, a visceral, gut-level feeling of what should be evaluated.
Planning and evaluation are interrelated, flip sides of the coin. Knowing where problems in implementation might occur is useful for targeting limited evaluation resources. Evaluators may be too far removed from what is being evaluated, objectivity is fine but the distance may be too great and insight may suffer.
Perceptions of a curriculum devoted specifically to evaluation
It was demanding to develop and teach theory, evaluation design, and other areas but without some of this the preparation for evaluators and evaluation as a field would suffer.
A ‘theory and nature of the field’ course is absolutely essential for evaluation as a field, a sine qua non condition. Even now, my sense is that a subset of evaluators do not have a solid grounding about what evaluation is.
Concerns unfortunately linger as to what the content and structure of evaluation preparation should be in graduate education and the relationship to recent forms of training such as the Evaluators’ Institute and AEA Professional Development Sessions. Numerous studies on what is being offered and the skills needed by evaluators have been done in the last 15 years (King & Stevahn, 2015).
The 2015 New Directions in Evaluation Issue (Altschuld & Engle, 2015) is relevant here.
Quantitative methods (and why multiple methods are clearly necessary)
Quantitative training in the field came from a traditional base and was utilitarian.
Generating items for varied types of instruments stood me in good stead during my work life.
This emphasis of programs in the 1960s-1970s was valuable, however, it underscored a weakness that still remains, and, requires attention currently. Again, strong quantitative skills and understandings are important.
Table 3
Areas of Weakness in Understandings From Training and a Career
Area Weakness Comments/Discussion
Lack of qualitative methods
Qualitative methods were virtually non-existent in my preparation. Programs and innovations were thought of years ago from a mechanistic view of how they operate.
Absence of the view that programs are living collectives with dynamic, interpersonal dimensions and political forces at play and lurking ominously under the surface.
It seemed that if one knew quantitative strategies it would be easy to do interviews, observations, small group techniques, open-ended questioning, etc.
“They are just soft methods, what does a person really need to know.” Constructing scaled instruments is not the same as framing and leading a thoughtful, probing interview. Subtle personal and political factors influence evaluations and evaluators, being alert to them is vital.
Lack of knowledge of the breadth of qualitative methods
Tied to the above entry is that a range of methods (nominal groups, the Delphi technique, cyber focus groups) are now available to evaluators.
Early training did not prepare evaluators for getting into the subtle innards of programs.
With the passage of time and new options, another set of problems is being encountered:
- Use of mixed methods; - Presenting findings from mixed
methods;
- Selecting optimum methods; - Interpreting data from multiple
sources and methods; - Others.
Awareness in such areas would help evaluators.
Generalist or specialist
My program was a good for my career and its benefits are not to be minimized. The breadth of what I received was useful but left gaps in depth of knowledge and skills.
In my career I relied on others to fill in gaps with my background rounding out what they brought to the table. Many well received publications and products with co-workers have been generated, but my grounding might have been stronger.
It may not be possible to be trained in evaluation (history, models, theory, research studies, etc.) and at the same time be versed in measurement, statistics, and/or qualitative methods. Evaluators in fields like education, public health, social work, public policy, etc. must have understanding of substantive issues.
What is the appropriate balance between deep and breadth, how should graduate programs be organized, where do they fit into the preparation picture?
- What do they need and at what level do they need it? - What are the best ways to prepare people for the job?
- What should university programs consist of and are there sufficient resources (staff and monetary support to deliver the goods and will they ever have it)?
- Should we train individuals for practice and at the same time those who will build the research base of evaluation? (These are equally valuable for the field)
- What should be the relative weights of university venues versus others for preparing evaluators (institutes, AEA professional development offerings, and so forth for educating professionals?
Let us now turn to Question 3. What has been the impact of change?
Question 3: How Has the Field Changed Over Time?
In the United States, evaluation has evolved slowly and gradually at times, metamorphically and strikingly at others. Three areas, chosen based on my familiarity with them, will be highlighted. They are professionalization of the field, the preparation of evaluators, and needs assessment.Professionalization of Evaluation
A remarkable trend affecting evaluation is the dramatic growth of organizations not just in the U.S. but across the globe. In 1979, I attended my first national conference (the Evaluation Network and approximately 200 people were in attendance. For the next 6 years I went to joint meetings of the Evaluation Research Society and the Network until 1986 when they merged into the American Evaluation Association (AEA) and things began to mushroom. It has become a major group in the world but not the only one by far. There are the Canadian Evaluation Society and national and regional organizations in Europe, Asia, and elsewhere. In conjunction with AEA are many local affiliates, collectively they offer an organizational home for evaluators. It is just not numbers; it is what is now accessible to evaluators. A partial list is given below:
- A large overall community for the exchange of ideas, strategies for implementation, and avenues for research;
- More than 10,000 plus members worldwide where in prior times there were not groups whose singular focus was the field of evaluation; - At AEA and in local or regional venues, forums and training sessions
are routinely provided for upgrading and improving what evaluators do;
- AEA provides over 60 professional development training classes at the annual conference. They have been a large part of its activities for a long time;
- The 2015 AEA national conference had more than three thousand attendees on site and 1500 or so participating on-line for a total of about 5,000 (at ENET in 1979, 200 were there);
- International attendees at U.S. national conferences with friendships and connections being created on a world scale;
- A set of procedures for credentialing evaluators (titled as Designations) generated and conducted by the Canadian Evaluation Society;
- Major evaluation journals of high quality devoted to the practice of evaluation and its empirical and theoretical foundations.
The evaluation environment cuts across settings and cultures. In the U.S. there is a diverse community of practitioners, much more so than in the formative years when academics predominated. The scene has also shifted from the insular stance of U.S. evaluators to a more wide-ranging view.
The Training of Evaluators
Periodically, AEA and predecessor groups have commissioned studies of programs for the preparation of evaluators, mainly in universities. I had a major role in two of them (Altschuld, Engle, Cullen, Kim, & Macce, 1994; Engle & Altschuld, 2003/2004; Engle, Altschuld, & Kim, 2006). LaVelle has conducted several recent investigations and I have been honored to be in communication with him about them. He found that programs having two or more courses devoted to evaluation content (a definition specified in 1994) had increased in the last few years. This finding came from an electronic search of courses in university bulletins from many countries and categorizing titles and topics covered. The results are intriguing but important questions could be raised:
1. How often are they taught? 2. What students are participating?
3. What are the full time equivalents for teaching?
4. Are some more in the realm of service for disciplines such as public health or social work as opposed to the preparation of individuals whose careers are fully in evaluation?
5. Is a course with research and evaluation in its title really evaluation focused?
6. What is the background of the instructors?
Other observations from his investigation raise serious concerns for evaluation.
1. He was not able to identify one course on needs assessment which informs the design of programs and their formative and summative evaluation.
2. There were almost no courses related to qualitative methods. That is disappointing because of their importance for understanding what is being evaluated.
Later, LaVelle and Donaldson (2015) noted strategies for preparing evaluators including on-line offerings, institutes, distance education, etc. Still, it is appropriate to ask how the field should handle the preparation of those who will carry its flag into the future. For a moment return to the description of my training. More options are now available for acquiring skills and understandings which is exciting. But they come with the cost of more complexity and the concern of how new opportunities might be balanced with what graduate training can provide?
Needs Assessment (NA)
Needs assessment was selected as a topic for this section not only because of my involvement with it but additionally because it is undergoing rapid change and has mutated from a technical activity to a more organic one. What does that mean? In early writings about the process methods and models (a sterile type of thinking) were stressed. Kretzmann and McKnight (1993) harshly criticized this stance in their seminal book on “Building Communities from the Inside Out” which led to some assessors using assets not needs to start planning programs in community development and public health. Needs were not as key in guiding planning and development efforts. Asset based approaches rely more on integrating affected stakeholders into them. As time
passed, a hybrid (Altschuld, 2014) for conducting joint asset/capacity building and needs assessment (NA) endeavors was proposed.
Many examples are in the recent literature. A hybrid is more complex and difficult to implement. It takes more time and skills to do the investigation, and the cost will be higher than for NA or asset identification by itself. The argument is that either process apart from the other will fall short of providing solutions to major issues like drugs, poverty, violence, public health concerns, etc.
Moreover, doing both is not to be implemented by an outside consultant who is there to direct the endeavors for communities, agencies, or organizations but rather who has to consider individuals in the situation as full partners in activities and resultant changes. This will complicate what needs assessors and asset capacity builders do and require them to include groups and their members as co-travelers in the journey of exploration—Everyone has a stake and vestment in it. Hopefully this will promote better outcomes. These shifts are subtle. The person working with the organization becomes a guide and facilitator, not a controller. This transformed role necessitates patience and fortitude for the practitioner and should produce an outcome that all players respect as meaningful and coming from collective participation. Buy-in to solutions should be greater.
Now we turn to conclusions and insights that might be drawn from the brief historical trip down memory lane. What does it mean for the future of evaluation?
Conclusions
First, the rate of change of evaluation will continue and be more astounding in the next decade, given what has occurred in the past. Judging by how evaluation has taken off in professional groups and international connections, the growth of preparation programs, the effects of technology, newer perceptions of NA and asset capacity building as illustrations, the field is on a fast, expansive, and upward course. Where this winds up is anyone’s guess but for the foreseeable future, evaluation is on the rise.
In this vein it would be useful to know the nature of career pathways of established evaluators and new entrants to the field. What do they actually do on their jobs, what prior learning do they utilize and what do they not use, what do they value, what is the nature of the workplace, what are the long-
term job prospects, how do evaluators advance in jobs, and so on? We need to better understand what evaluation is and what is going on in the field?
Second, with the growth in preparation opportunities in universities and other venues and the offering of professional designations for evaluators, we need to specify what it means to be an evaluator and the prerequisite skills and background ‘designated’ individuals should have. It is imperative to study what is and isn’t being taught in graduate schools as well as in other settings. What content is being covered, are there critical skill or knowledge areas missing, what is or should be there at minimum and higher levels, what continuing education would be required, what kinds of experiences should be embedded in training, over what time period would it take to develop individuals for entry types of designations and higher ones?
There are more fundamental and deeper concerns inherent in this second conclusion. If we have a field, what should practitioners understand as to its past and evolution? What about history and early development? Shouldn’t all evaluators have some bedrock knowledge? What is the balance for the profession, if the focus is solely on practice and practitioners what does that say about the research and the theoretical base of the enterprise?
Not having a balance might not matter in the short run but in the long term the consequences could be serious. Will evaluation become a collection of techniques and methods instead of a field with a unified foundation? Generating articles about theory, methods, models for practice, and research on practice is important and should be encouraged and supported. If we weigh too heavily on the conduct of what is done and away from the ‘whys’ then the concern is about the viability of the field. In fact, will designating and/or credentialing evaluators possibly deemphasize the role of research and contributions to the evaluation literature?
I made the point earlier that not enough practitioners know the origins of the field, its development, and related matters. My observations from attending presentations at AEA for 30 plus years and at other conferences are that presenters can tell you what they did but often they cannot specify or possibly don’t know what might be wrong with their methods or practice. For example, many practitioners are implementing NAs but I often want to ask:
- What might be wrong with your methods,
- Where might your findings be suspect or even be invalid, - What alternatives were considered and ruled out, and - What was the reasoning underlying not using them?
I suspect that responses would be thin. My perception is that evaluators and those specializing in NA did not have the basic experiences in planning programs as in my graduate training. This aspect of evaluator preparation might require closer scrutiny. That same thinking would be for history, evaluation models, and so on. What are the theoretical underpinnings of evaluation every evaluator ought to understand?
Analogously, the same type of thinking would be relevant to qualitative methods. This might include how to interview, subtleties in observing phenomena, ways to interpret evidence, dealing with touchy cultural issues, and evaluator biases as they might affect results.
The third conclusion is that technological development will continue to have a profound effect on our work. Although not explicitly discussed in the previous text, it was there between the lines. Watkins and Altschuld (2014) speculated on what this could mean for needs assessment. Assessors might take advantage of social media to get information to involved stakeholders or gather information from them or employ Photovoice to learn about the strengths and weaknesses in a local environment from photographs. An example of the technique is to have community members take photos of strengths and weaknesses they see and then they may even do most of the interpretation of what they mean. Since cell phones are common, this might be done with minimal cost.
Data could be collected in real time. Participants in studies could use phones to provide input on an almost immediate basis as has already been used in one national health study with a huge sample size. More and more, large and well maintained data bases (big data) exist. If they fit what an assessment is about then by all means use them, they may be free with the only expense being for analysis. The NHANES data base is one that is available. It was built on a statistically sound national sample of youth ages 5-19 and contains a vast array of health oriented variables that were carefully measured. It was federally funded in the U.S. with no charge to interested parties.
Geographic information systems (GIS) are an old idea that has undergone a metamorphosis with the advent of new technology. I recall seeing a NA article using geographic principles perhaps 30 years ago. It was interesting but by today’s standards it would look primitive. (See Hites et al., 2013, for a demonstration of how geographic and mapping procedures were employed to identify crime at a large college campus in a big city.) It isn’t a stretch to see the influence of technological advancement on what we do. In fact, the only limitation here may be our imagination.
Final Thought
This excursion through history was not intended to be comprehensive. The purpose was to get evaluators and others interested in evaluation to be contemplative about what and why we do things, and what the field might become as it moves toward the status of a profession. If that has happened then it was successful and thanks for joining in on the journey.
References
Altschuld, J. W. (2014). Bridging the gap between asset/capacity building and needs
assessment: Concepts and practical applications. Thousand Oaks, CA: Sage.
Altschuld, J. W., & Engle, M. (2015). The inexorable historical press of the developing evaluation profession. In J. W. Altschuld & M. Engle (Eds.),
Accreditation, certification, and credentialing: Relevant concerns for U.S. evaluators: New directions for evaluation (No. 145, pp. 5-19). San Francisco, CA:
Jossey-Bass.
Altschuld, J. W., Engle, M., Cullen, C., Kim, I., & Macce, B. R. (1994). The 1994 directory of evaluation training programs. In J. W. Altschuld & M. Engle (Eds.),
The preparation of professional evaluators: Issues, perspectives, and programs. New directions for program evaluation (Vol. 1994, No. 62, pp. 71-94). San
Francisco, CA: Jossey-Bass.
Engle, M., & Altschuld, J. W. (2003/2004). An update on university-based evaluation training. The Evaluation Exchange, 9(4), 13.
Engle, M., Altschuld, J. W., & Kim, Y. C. (2006). 2002 Survey of evaluation preparation programs in universities: An update of the 1992 American evaluation association-sponsored study. American Journal of Evaluation, 27(3), 353-359. Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation:
Alternative approaches and practical guidelines. Upper Saddle River, NJ:
Pearson.
Hites, L. S., Fifolt, M., Beck, H., Su, W., Kerbawy, S., Wakelee, J., & Nassel, A. (2013). A geospatial mixed methods approach to assessing campus safety.
Evaluation Review, 37(5), 347-369.
King, J. A., & Stevahn, L. (2015). Competencies for program evaluators in light of adaptive action: What? So what? Now what? In J. W. Altschuld & M. Engle (Eds.), Accreditation, certification, and credentialing: Relevant concerns for U.S.
evaluators: New directions for evaluation (No. 145, pp. 21-37). San Francisco,
CA: Jossey-Bass.
Kretzmann, J. P., & McKnight, J. L. (1993). Building communities from the inside out:
A path toward finding and mobilizing a community’s assets. Chicago, IL: ACTA.
LaVelle, J. M. (2014). An analysis of evaluation education programs and evaluator
skills across the world (Unpublished doctoral dissertation). Claremont Graduate
University, Claremont, CA.
LaVelle, J. M., & Donaldson, S. I. (2015). The state of preparing evaluators. In J. W. Altschuld & M. Engle (Eds.), Accreditation, certification, and credentialing:
Relevant concerns for U.S. evaluators: New directions for evaluation (No. 145,
pp. 39-52). San Francisco, CA: Jossey-Bass.
Stufflebeam, D. L. (1973). Educational evaluation and decision-making. In B. R. Worthen & J. R. Sanders (Eds.), Educational evaluation theory and practice (pp. 128-149). Belmont, CA: Wadsworth.
Watkins, R., & Altschuld, J. W. (2014). A final note about improving needs assessment research and practice. New Directions for Evaluation, 2014(144), 105-114.
Worthen, B. R., & Sanders, J. R. (1973). Educational evaluation: Theory and practice. Belmont, CA: Wadsworth.
