應用行動模式/改變模式之圖式規劃
臺灣的學習共同體方案
潘慧玲
陳惠次
摘 要
評鑑通常在促進方案改進的歷程中是最後一個步驟,評鑑結果 可以提供方案計劃者與利害關係人了解方案缺失,以便在下一階段 謀求改善。不過,近來在設計階段即運用評鑑工具與原則以改進方 案計劃的歷程上,有日益繁增的興趣。在本文,我們討論了一個較 為新近發展的評鑑工具-行動模式 改變模式,作為引導臺灣一項 大型教育方案-學習領導下的學習共同體(簡稱學習共同體方案) 的規劃,同時也作為評鑑架構之用。我們組織了工作小組會議發展 學習共同體的行動模式 改變模式之圖式(schema)。在會議中, 參與者討論了圖式中的要素、要素與方案的關連,以及從圖式的發 展中所習得的經驗。而針對會議中蒐集的質性資料,採內容分析法 進行分析。經過整個圖式發展的過程後,參與者表示行動模式 改 變模式之圖式提供了一個討論與發展方案計畫的有效平台,激發了 批判性思考,有助於確認方案的主要要素,且提供了評鑑設計一個 整全性的引導。該圖式是一個較為新近發展的工具,能俾利評鑑者 與利害關係人清楚描繪方案或設計方案,只是目前在少數發表的論 文中僅見其應用於公共健康方案。故而,本研究所獲得的結果,一 潘慧玲(通訊作者),淡江大學教育政策與領導研究所教授 電子郵件:panhlw@gmail.com 陳惠次,美國 Mercer University 公共衛生系教授 電子郵件:chen_h@mercer.edu 投稿日期:2016 年 5 月 28 日;修正日期:2016 年 10 月 3 日;接受日期:2017 年 1 月 11 日 《研究紀要》方面可提升教育與其他方案使用該圖式進行方案計劃與評鑑設計, 另一方面對於未來在評鑑計劃工具的運用上,亦有鼓勵作用。
APPLYING THE ACTION MODEL/CHANGE
MODEL SCHEMA IN PLANNING THE
LEARNING COMMUNITY PROGRAM
IN TAIWAN
Hui-Ling Wendy Pan Huey T. Chen
ABSTRACT
Evaluation is often the final step in running social betterment programs. For improvement, evaluation findings inform program planners and stakeholders about program shortcomings. However, interest has recently grown in applying evaluation tools and principles to programs from the design stage, to improve planning. We discuss the application of a relatively new evaluation tool, the action model/change model schema, for guiding the planning of a large-scale education program, the Learning Community under Leadership for Learning (Learning Community Program) in Taiwan and serving as a framework for evaluation at the same time. We organized work-group meetings for the development of an action model/change model schema for the Learning Community Program. The participants discussed the components of the schema and how they related to the program, as well as the lessons learned from the experience of developing the schema. Content analysis was applied to the qualitative data collected during the work-group meetings. The participants indicated that the schema provided an effective platform for discussing and developing the program plan. The schema also stimulated critical thinking, helped identify the major components of the program, and provided a holistic guide for the evaluation design. The action model/change model schema is a relatively new tool used by evaluators to assist stakeholders in describing or designing a program more effectively. Few published articles discuss its applications in public health programs. This study investigated the
Hui-Ling Wendy Pan (corresponding author), Professor, Graduate Institute of Educational Policy and Leadership, Tamkang University, Taipei, Taiwan.
E-mail: panhlw@gmail.com
Huey T. Chen, Professor, Department of Public Health, Mercer University, Macon, U.S.A. E-mail: chen_h@mercer.edu
Manuscript received: May 28, 2016; Modified: October 3, 2016; Accepted: January 11, 2017
application of the schema to an education program outside the United States. The lessons learned from this application can promote the schema as an effective tool for planning and designing the evaluation of education and other programs, as well as encourage the use of evaluation planning tools for strengthening program designs in the future.
Keywords: action model, change model, program evaluation, leadership for learning, learning community
Introduction
An intervention program consists of multiple components and linkages. Often times, communicating this information to people inside and outside of the program is difficult. Stakeholders have been increasingly asking for assistance from evaluators, in order to strengthen their program plan and to better describe and/or communicate their program. The evaluation community responded to this demand seriously as the theme of the 2016 Annual Meetings of the American Evaluation Association was “Evaluation Design”. The event challenged evaluators to expand the traditional scope of evaluation from
implementation and outcome assessments to program plan assessment.
This paper attempts to discuss experiences and lessons learned from applying an evaluation tool, the action model/change model schema, to planning a large-scale education program in Taiwan, called the Learning Community under Leadership for Learning Program (further referred to as the Learning Community Program). The program involves dynamic processes with multiple-level interventions and outcomes, making it difficult for stakeholders to fully comprehend, effectively communicate, and adequately evaluate it. A member of the program team proposed to select an evaluation-planning model to help describe the program and to provide a framework for evaluation. After conducting a literature review, three models were under consideration: Logic models, the causal loop diagram, and the action model/change model schema.
The Logic Model is a popular tool used to describe an intervention program through a graphic representation of the relationships between the following key-components: (1) Inputs (i.e., resources dedicated to, or consumed by the program); (2) Activities (i.e., what the program does with the inputs to fulfill its mission); (3) Outputs (i.e., the direct products of program activities); and (4) Outcomes (i.e., benefits to participants during and after program activities). The logic model includes arrows connecting Inputs to
Activities, Activities to Outputs, and Outputs to Outcomes (United Way of
America, 1996; Centers for Disease Control and Prevention, 1999). The application of the logic model (Julian, Jones, & Deyo, 1995; Kaplan & Garrett, 2005; Knowlton & Phillips, 2013; McLaughlin & Jordan, 1999; Renger & Titcomb, 2002) indicates that it is useful for identifying major program components and indicators for communication and evaluation purposes. However, critics pointed out that the logic model typically prescribes a linear, one-way direction from one component to another (Sabatier, 1999; Reeler,
2007); It may not be suitable for programs which are dynamic or have multi-level interventions.
The causal loop diagram is another popular tool introduced by proponents of systems thinking, for describing and explaining complex relationships within an intervention program (Sterman, 2000; Midgley, 2003; Peters, 2014). Systems thinking argues that a system where components tend to interact with each other cannot be understood by reducing it to smaller parts and components. The causal loop diagram completes the model by picturing feedback loops indicating the existence and direction of a causal relationship between components or variables (the plus/minus sign indicates that two variables change in the same/opposite direction). Although causal loop diagrams allow a better representation of dynamic relationships within a program, the literature reports difficulties with data analysis, interpretation of findings, and communication of results when using a causal loop diagram. For example, the UK government’s Foresight Programme (Butland et al., 2007; Savigny & Adam, 2009) applied systems thinking to understand the complex relationship between biological and social determinants of obesity. The causal loop diagram contained 108 variables and their relationships illustrated with more than 300 lines (indicating positive and negative influences). These connections ranged from between two to as many as 16 variables. It was unclear how to analyze the large number of causal loops by using exiting statistical techniques, how to summarize and communicate the results in a meaningful way, and how to draw evaluation conclusions about the effectiveness of the program.
The action model/change model schema integrates contextual factors and causal mechanisms in the intervention process (Chen, 2015). The schema represents the program as a systematic configuration of the stakeholders’ prescriptive and descriptive assumptions underlying programs, whether explicit or implicit. The prescriptive assumptions (the action model, or how to do it) explain what actions must be taken in order to activate the change model to produce desirable outcome. The descriptive assumptions (change model or why it works) explain the causal processes expected to happen in order to attain program goals. The action model/change model schema presents the components of multi-level interventions and their relationship in a simple way, facilitating a view of the program in entirety. This particularity makes it easy to spot weak links in the program, thus making the schema a useful planning tool, and identifies the pathways of reaching outcomes, thus also making it a
useful evaluation tool. However, it is relatively new and there are only a few cases illustrating its application in the United States (Chen, 2015).
The project team of the Learning Community Program selected the action model/change model schema because it matched the program’s intention well. However, the team acknowledged its major limitation, the lack of enough examples to guide its application. To overcome this limitation, the project team solicited assistance from the schema developer; he agreed to serve as a voluntary consultant. This was the first time of using the schema for planning and evaluating education interventions. This study discusses the process and lessons learned from applying the action model/change model to the Learning Community Program in Taiwan.
The Learning Community Movement and the Learning
Community Program in Taiwan
Theoretical Foundation of the Learning Community Movement
The concept behind learning community is that people with common attitudes and goals can improve their teaching skills and ultimately the academic performance of their students if they meet regularly, share expertise, and work collaboratively (Spillane & Camburn, 2006). There is evidence that implementing learning community in schools has the potential to revitalize school teaching and to prepare competent students for the future (Polly, Heafner, Chapman, & Spooner, 2015; Pont, Nusche, & Moorman, 2008; Robinson, Lloyd, & Rowe, 2008).
Student learning is not a stand-alone activity, but is closely connected with organizational conditions (Robinson et al., 2008). School leadership can improve learning by supporting and developing teacher quality, defining goals, measuring progress, strategically managing resources, and collaborating with external partners (Pont et al., 2008). Learning community uses broad-based learning to create the necessary environment for schools to become self- created agencies in which relationships among people are interconnected (Pan, 2014). There are five key features of learning community: Shared values and vision, collective responsibility, reflective professional inquiry, the promotion of group learning, as well as individual learning (Bolam et al., 2005).
Furthermore, Sato (2012), a Japanese scholar, who integrated western theories and local practices, proposed the ‘learning community’ (xue xi gong tong ti 學 習 共 同 體 ) as an approach to transform schools. Building
collegiality among teachers as well as constructing the classrooms as learning communities are the two main tasks. Teacher collegiality is featured as a cycle of: Teachers working together to plan the lessons, teaching the lesson by one teacher and the others observing, and discussing the lesson taught based on the data collected (Lewis, Perry, & Murata, 2006). Learning community emphasizes leadership, community building, and teacher development as points of inquiry and reflection. It is the responsibility of teachers to establish an environment conducive of dialogue and to encourage peer collaboration. Based upon the above stated principles, learning community can be implemented at the school, teacher, and/or student level. Regardless, participants need to be willing to share, reflect, dialogue, and cooperate to enhance performance. Teachers feel motivated by the premise of power- sharing and democratic decision-making in the learning process. The concept of learning community is so appealing to educators that many countries, including Taiwan, have adopted it as an approach to reform the education system.
The Learning Community Program in Taiwan
Echoing local demands as well as education reforms in the global context, the Taiwan Ministry of Education has launched a series of policies to revitalise school teaching and to prepare students’ competence for the new century. A policy promoting professional learning community in schools was implemented in 2010, to encourage staff to work collaboratively. Furthermore, the notion of transforming the school as a learning community advocated by Sato, was introduced to Taiwan. With more than 30 years of practices, Sato indicated that his approach was applied in over 3,000 schools, and it became a silent revolution of learning in Japan (Sato, 2010, 2012). Nowadays, ‘learning community’ (LC) has become a buzzword in Taiwanese schools, and was tested out in several schools.
To indigenize Sato’s approach of learning community in Taiwan, leadership for learning (LfL) was proposed as a superordinate concept of LC (Pan, 2014; Pan, Lee, Hwang, Yu, & Hsueh, 2014). Marsh, Waniganayake, and Gibson (2014) explored the concept of leadership as an organizational behaviour and suggested LfL as a vehicle for community engagement. In addition, leadership is not viewed as position based (Leithwood, Harris, & Hopkins, 2008; Spillane, 2005), but also inherently relational and interactive (Day, 2011). A milieu of diverse interactions where teachers support,
challenge, and learn from each other, fosters agency and school capacity. When embracing a constructivist view of teaching and learning, as Sergiovanni (2001) suggested, the emphasis is on leadership as community building, and teacher development as inquiry and reflection. By integrating the perspective of distributed leadership, LC becomes an environment where leadership practices manifest.
Funded by the Ministry of Education, an indigenous model, the Learning Community under Leadership for Learning Program (also referred to as the Learning Community Program), was launched in 2013. The program promoted three components: (1) building the school as the learning community; (2) constructing teacher learning community; (3) creating the classroom as learning community by executing learning-centered pedagogy. In the year of 2016, 33 schools enrolled (15 of them are primary schools); and a total of 737 teachers and 10,262 students participated in the Program. Handbooks introducing the indigenous conceptions and practices were developed (Pan, Lee et al., 2014; Pan, Hwang et al., 2015; Pan, Lee, Hwang, Yu, & Hsueh, 2016).
The Action Model/Change Model Schema
The project team used the conceptual framework of the action model/change model schema (Chen, 2015) to describe the Learning Community Program plan. We are introducing the schema before presenting the planning trajectory.
Change Model
A change model describes the causal process generated by the program. The elements of a change model consist of the following:
Goals and outcomes. Goals reflect the desire to fulfill unmet needs, such
as poor health, inadequate education, or poverty. Outcomes are the concrete, measurable aspects of these goals.
Determinants. To reach goals, programs must identify leverage
mechanisms upon which to develop a treatment or intervention. That mechanism is variously called the determinant, mediator, or intervening variable.
Intervention or treatment. Intervention or treatment means any activity
in the program that aims directly at changing a determinant. It is, in other words, the agent(s) of change within the program.
Action Model
An action model is a systematic plan for arranging staff, resources, settings, and support organizations, to reach a target group and deliver intervention services. The action model consists of the following elements:
The implementing organization. The implementing organization is the
entity coordinating the program, and is usually responsible to allocate resources, to coordinate activities, and to recruit, train, and supervise implementers and other staff. How well a program is implemented may be related to how well this organization is structured. Initially, it is important to ensure that the implementing organization has the capacity to implement the program.
Program implementers. Program implementers are the people
responsible for delivering services to clients such as counselors, case managers, outreach workers, school teachers, health experts, and social workers. The implementers’ qualifications, competency, commitment, enthusiasm, and other attributes can directly affect the quality of service delivery.
Peer organizations/community partners. Programs often may benefit
from, or even require, cooperation or collaboration between their implementing organizations and other organizations. If linkage or partnership with these useful groups is not properly established, implementation of such programs may be hindered.
Intervention and service delivery protocols. Intervention protocol is a
curriculum or prospectus stating the exact nature, content, and activities of an intervention—in other words, the details of its orienting perspective and its operating procedures. Service delivery protocol, in contrast, refers to the particular steps to be taken to deliver the intervention in the field.
Ecological context. Some programs have a special need for contextual
support, meaning the involvement of a supportive environment in the program’s work. Both micro-level and macro-level contextual support can be crucial to a program’s success. Micro-level contextual support comprises social, psychological, and material supports to ensure clients’ continued participation in intervention programs. Macro-level context includes community norms, cultures, and political and economic processes.
Target group (to be identified, recruited, screened and served).
Crucial elements necessary for selecting the target group are valid eligibility criteria, feasibility of reaching and serving the target group effectively, and
the willingness of potential clients from the target group to commit and cooperate with the program.
Figure 1 illustrates the relationships among the components of schema. The action model must be implemented appropriately to activate the “transformation” process in the change model. For a program to be effective, its action model must be sound and its change model must be plausible. If this is the case, implementation is likely to be doing well. Figure 1 also illustrates evaluation feedback represented in dotted arrows. Information from implementation can be used to improve the planning or the development of the action model. Similarly, information from the change model can be used to improve the implementation process and the action model.
Figure 1. Action Model/Change Model Schema
Methodology
The participants of this study were research team members of the Learning Community Program. With the assistance from the consultant, the participants developed the schema in three working-group meetings. School representatives were invited to attend the last meeting. During meetings, participants actively talked about the components of the schema, and their
relevance to the program. All conversations were recorded and later transcribed.
The research team used qualitative content analysis to analyze data (Krippendorff, 1980); they read the transcriptions, listened to the recordings several times individually, and analyzed the data separately. The team organized the participants’ statements into categories based upon semantic and conceptual similarity, and then abstracted major themes. The second author and a research associate met five times to discuss the analyses, categories, and themes until they reached a consensus. To enhance accuracy, the first author checked the analyses, categories, and themes to meet the criteria of peer reviewing (Creswell, 1998).
Major Issues of the Development of the Action
Model/Change Model Schema
The Learning Community Program is complicated and dynamic; the schema brought about intensive discussions regarding the mapping of the program. The discussions of the following major issues raised during the development of the schema were particularly informative and productive:
Differences Between Action Model and Change Model
Participants asked questions about differences between action model and change model, and whether schema allowed for straightforward identification of variables of interest for statistical analysis. For example, one of the members raised the following question:
In addition to assisting us in identifying what we’ve done, how can this action model be used? Can it be used as a research framework? If we need a research framework, we can just use the change model!
The developer of schema explained that the action model describes components related to the implementation of the program. These components function as contextual factors for the change model to work properly. The action model is particularly useful for guiding process evaluation. The action model energizes the change model for generating causal processes that link the intervention to the outcomes. Usually, the assessment of the action model uses mixed methods. The relationships between variables in the change model can be analyzed by applying statistical techniques such as path analysis or structural equation models.
Issues on Implementers Versus Target Population in the Program
Implementers are generally the people who provide services. Initially, the team identified the following two groups as implementers: (1) project staff and consultants at the university level and (2) school administrators and teachers at the school level. The Learning Community Program required training of both groups for delivering services, however, in different ways: Project staff and consultants at the university level trained to become trainers and mentors for school administrators and teachers, while administrators and teachers at the school level trained to develop knowledge and skills to apply the learning community in schools and classrooms. One participant stated:
Although the learning communities are put into practice in the school fields, our project staff launched the program. Our project staff also trains the school administrators and teachers to build learning communities.
Through discussion, implementers began to realize that the program was operating under the principles of Training-the-Trainers Model. That is, the project office first trained the staff and consultants as trainers of the learning community, whom in turn trained administrators and teachers to practice the learning community at schools. From this perspective, project staff and consultants were implementers and school administrators and teachers were the target populations for this intervention. School administrators and teachers became implementers after they trained to implement and practice the learning community. A description of their roles is included in the section of the change model.
Issues on Describing the Interventions and Change Processes
The action model/change model schema requires users to identify the interventions and the causal processes for attaining outcomes. Since the Learning Community Program consisted of three levels of interventions (schools, teachers, and students), the participants had intensively discussed how to reflect these causal processes in a change model. The discussions focused on the following two options:
Option 1. One of the options was to create a change model that included
all three levels of interventions in a single diagram, similar to the change model illustrated in Figure 1, inclusive of each intervention with its own
causal process and outcomes. The reasoning for this choice was that the program is implemented at three levels simultaneously.
Option 2. Another option was to create three change models, each
representing one level of intervention. One of the participants stated: It seems that the mutual influences among the different levels could not be captured by the diagram of Option 1. We might need to think of another way to do it.
During discussions, participants felt that Option 1 would not properly reflect the relationships across the different levels of change processes. The literature indicates that school-level changes must take place first, in order to support changes at the teacher-level; and teacher-level change must happen before student-level change can occur. On the other hand, the changes at the teacher- and student-level might cause transformation of the school. Option 1 could not capture these change processes. The team decided to adopt Option 2, which clearly illustrates that the school-level change model was a necessary condition that made the teacher-level change model and student-level change model possible.
Finalizing the Action Model/Change Model Schema
The final version of the action model/change model is illustrated in Figure 2:
Action Model
The action model consists of the following components:
Implementing organization. The project office was established for
coordinating program activities, such as hiring personnel, establishing partnerships, coordinating activities, and developing the intervention protocol.
Implementers. Implementers were staff and consultants responsible for
training and mentoring school administrators and teachers.
Associate organizations/partners. The project office built partnerships
with universities, government agencies, and schools. This included a provision of services to assist them in planning and implementing the program. Another effort in this component was to connect schools as a network of resources and information.
Ecological context. The project office and partners launched campaigns
to promote the Learning Community Program via conferences and media, to create a milieu for supporting the program.
Figure 2. Action Model/Change Model of the Learning Community Program
Intervention and service delivery protocols. The project office and
partners developed models to help adapt the notion of learning community such as it would embed to Taiwanese culture. Adaptation of the program made application more feasible in Taiwanese schools and communities.
Target populations. The project office assisted by partners was
responsible for recruiting schools and teachers to participate in the program.
Change Model
The implementation of the action model was expected to generate change processes at the school, teacher, and student level. The interacting elements producing change at each level were as follows:
School-level.
1. Intervention: Train and mentor school administrators for school- level interventions.
2. Determinants: Increased administrators’ competency and capability in initiating and practicing learning communities in their respective schools.
3. Outcomes: Administrators will create structural and policy changes for supporting the learning community activities in schools, and are likely to enhance the materialization of school as learning community featured with supportive and shared leadership, building vision and recognition, learning for change and shared personal practice.
Teacher-level.
1. Intervention: Train and mentor teachers for practicing learning communities.
2. Determinants: Increased teachers’ skills, knowledge, and
commitment for practicing learning communities.
3. Outcomes: Increased dialogues, collaboration, and experience sharing among teachers and increased capacity for professional development.
Student-level.
1. Interventions: Conduct learning-centered teachings in classrooms.
2. Determinants: Increased students’ engagement in inquiry,
collaboration, and expression as well as improved social interactions and relations in classrooms.
3. Outcomes: Increased students’ engagement of learning, enhanced learning power, and enhanced performance.
Relationships among the three levels. Figure 2 indicates how the three
levels of intervention are related. The school-level change was necessary first in order to support the teacher-level of change. Furthermore, both, school- level change model and teacher-level change model must take place before the student-level change model would work.
Impacts of the programs. Figure 2 shows how these three-levels of
changes created the following overall impacts: Improving student achievement and potential to pursue a career after graduation, increasing the number of schools adopting learning communities, and contributing to improving government education policies.
Lessons Learned in Applying the Action Model/Change
Model Schema
After the completion of the action model/change model schema for the Learning Community program, the participants further discussed their views and lessons learned from such activities. During this meeting, participants indicated that working with the action model/change model schema for the Learning Community Program helped them understand the program in its entirety. They were pleased to have developed a schema that effectively synthesized their multilayered intervention in a graphic format easy to understand and communicate. However, they advised that the schema is a collaborative process and requires all participants’ contribution in terms of time and effort. Based upon this experience, the research team summarized the benefits and limitations of the schema as follows:
Benefits of Applying the Action Model/Change Model Schema
The schema provided a platform for a participatory discussion in developing the program plan. The schema provided a comprehensive
platform for a participatory approach in which team members responsible for different tasks could work together. This process increased the participants’ commitment and support of the program. One participant said it best during the last meeting:
I was unclear about how my task was relevant to others. The discussion and application of the schema to the program was very helpful in understanding how my work relates to others’ endeavors and how the overall goals of the program can be achieved.
The schema provided deeper insight into the program. The schema
required stakeholders to clarify contextual factors and causal mechanisms. This task involved deep reflection about the program, and about what was necessary to ensure that the program works. As provided in this study, these reflections generated new insights as well as a new understanding of the program. Another meeting participant stated:
I was impressed with the schema’s capability for nicely describing different layers of the program and how they were connected to each other for producing impacts. I feel that I have a better understanding of the program now.
The schema identified the major components of the program and provided a guide for evaluation. The schema provided a framework for
stakeholders and evaluators to effectively identify major components of the program (Chen, 2015). The schema also served as a useful foundation for communicating program information and for designing the evaluation, once all components and their relationships were clearly articulated. One of the participants stated: “the schema provides a systematic way to do process and outcome evaluation.”
Limitations of the Schema
It takes time and effort to understand and learn how to apply the schema to the program. The concepts of action model, change model, and
components within each model were new to participants. It took considerable time and effort for them to learn. One of the participants expressed that:
Some of the components in the schema are challenging for me to understand. For example, it took me quite a while to understand the component of determinant, and how is it different from the component of ecological context.
More examples of schema application would make it easier to apply.
The literature only provides two cases of applying the schema. Participants felt that more examples would be helpful. One participant stated:
The schema looks like a useful tool. However, since the schema is new to us, more examples especially in the education area are needed to help us understand its application.
Discussion and Conclusion
This study investigated the process of applying the action model/change model schema to describe, clarify, and strengthen the program plan of the Learning Community Program in Taiwan. The participants acknowledged that the schema provided a model that allowed them to understand and effectively communicate details related to the program. Furthermore, they stated that the action model provided a guide for designing and conducting process evaluation. Similarly, the change model provided a basis for conducting
theory-driven outcome evaluation. This study provided detailed discussion of the technical issues that may arise in the application of schema, and how to resolve difficulties.
Based upon the experience from this study, the authors make the following suggestions for future applications of the schema:
Start with the Action Model Portion of the Schema First
In a typical schema development meeting, evaluators and stakeholders have an option for starting the action model or the change model first. Lessons learned from this study indicate some benefits of starting with the action model first. Stakeholders of the Learning Community Program mastered the concept of the action model faster than the change model. Since the components of the action model reflects what stakeholders are doing on a day- to-day basis, they quickly grasp the concept and engage in discussions that contribute to developing the action model. The change model was more challenging. Stakeholders understood the concepts of interventions and outcomes, but had difficulties grasping the concept of determinants. It took more time and effort to figure out what are determinants, and which were relevant to their program. Stakeholders would be frustrated with the schema if they started working on schema with the change model.
Revisions to Action Model Based on Change Model Components and Vice Versa
When developing the change model of a schema, contradictions or inconsistencies may occur in the action and change models. In order to maintain coherency between models, often time revisions must be made. The process of developing the models will reveal reciprocal influences. Changes may be necessary and first attempts at the schema are rarely final drafts. As learned from the Learning Community Program, it took a few revisions to develop a satisfactory version of the schema.
Consider Adding Multiple Action Models and/or Change Models as Needed
This study demonstrates the versatility of the schema to address complex programs, by allowing the development of multiple action models and/or change models for a complete representation of all program components and interactions between them. In 2013, during a panel at the American Evaluation
Association annual meeting, the schema developer discussed the potential benefit of adding multiple action models and/or change models within a typical schema (as illustrated in Figure 1); however, no empirical basis existed at that time for this idea. This study provided evidence supporting the usefulness of a schema with multiple change models. For example, the Learning Community Program had three levels of interventions and outcomes. In order to address all layers, the basic schema was expanded to three change models as illustrated in Figure 2. Similarly, the action model can be expanded if needed. This study may inspire evaluators to apply the multiple action models and/or multiple change models for addressing complicated issues within intervention programs in the future.
In conclusion, this study illustrated the application of schema as a useful evaluation-planning tool in education, lending hope for potential application in other types of interventions. The action model/change model schema can help untangle components and relationships occurring in multilevel interventions. The process of developing a schema stimulates critical thinking and development of new ideas, promotes understanding and clarification of stakeholders’ roles and responsibilities, clarifies environmental factors modulating interventions, identifies leverages and mechanisms predicting intervention success or failure, and ultimately stimulates collaboration and involvement of all stakeholders. With the growing interest for the use of evaluation models and tools in the planning process, this study indicates that the schema can go beyond its evaluation function by applying it as a useful planning tool for designing intervention programs.
References
Bolam, R., McMahon, A., Stoll, L., Thomas, S., & Wallace, M., Greenwood, A., …Smith, M. (2005). Creating and sustaining effective professional learning communities. London, England: University of Bristol.
Butland, B., Jebb, S., Kopelman, P., McPherson, K., Thomas, S., Mardell, J., & Parry, V. (2007). Foresight tackling obesities: Future choices. London, England: Government Office for Science. Retrieved from
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/28 7937/07-1184x-tackling-obesities-future-choices-report.pdf
Centers for Disease Control and Prevention (1999). A framework for program evaluation in public health. Atlanta, GA: Author. Retrieved from https://www.cdc.gov/mmwr/PDF/rr/rr4811.pdf
Chen, H. T. (2015). Practical program evaluation: Theory-driven evaluation and the integrated evaluation perspective (2nd ed.). Los Angeles, CA: Sage.
Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage.
Day, C. (2011). The layering of leadership. In J. Roberston & H. Timperley (Eds.), Leadership and learning (pp. 13-26). London, England: Sage.
Julian, D. A., Jones, A., & Deyo, D. (1995). Open systems evaluation and the logic model: Program planning and evaluation tools. Evaluation and Program Planning, 18(4), 333-341.
Kaplan, S. A., & Garrett, K. E. (2005). The use of logic models by community-based initiatives. Evaluation and Program Planning, 28(2), 167-172.
Knowlton, L. W., & Phillips, C. C. (2013). The logic model guidebook: Better strategies for great results (2nd ed.). Thousand Oaks, CA: Sage.
Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Newbury Park, CA: Sage.
Leithwood, K., Harris, A., & Hopkins, D. (2008). Seven strong claims about successful school leadership. School Leadership & Management, 28(1), 27-42. Lewis, C., Perry, R., & Murata, A. (2006). How should research contribute to
instructional improvement? The case of lesson study. Educational Researcher, 35(3), 3-14.
Marsh, S., Waniganayake, M., & Gibson, I. W. (2014). Scaffolding leadership for learning in school education: Insights from a factor analysis of research conducted in Australian independent schools. Educational Management Administration & Leadership, 42(4), 474-490.
McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: A tool for telling your program’s performance story. Evaluation and Program Planning, 22(1), 65-72. Midgley, G. (Ed.) (2003). Systems thinking. Thousand Oak, CA: Sage.
Pan, H. L. W. (2014). School practices of leading learning in Taiwan. Leading and Managing, 20(2), 27-42.
Pan, H. L. W., Lee, L. J., Hwang, S. S., Yu, L., & Hsueh, Y. T. (2014). Learning community under the leadership for learning version 1.1: Basic handbook. Retrieved from https://sites.google.com/site/learningcommunityintw/lchandbook (In Chinese)
Pan, H. L. W., Hwang, S. S, Lee, L. J., Yu, L., Liu, S. M., & Hsueh, Y. T. (2015). Learning community under the leadership for learning version 2.0: Advanced handbook. Retrieved from
https://sites.google.com/site/learningcommunityintw/lchandbook (In Chinese) Pan, H. L. W., Lee, L. J., Hwang, S. S., Yu, L., & Hsueh, Y. T. (2016). Learning
community under the leadership for learning version 1.2: Basic handbook. Retrieved from https://sites.google.com/site/learningcommunityintw/lchandbook (In Chinese)
Peters, D. H. (2014). The application of systems thinking in health: Why use systems thinking? Health Research Policy Systems, 12(51), 1-6.
Polly, D., Heafner, T., Chapman, M., & Spooner, M. (Eds.) (2015). Professional development schools and transformative partnerships. Hershey, PA: IGI Global. Pont, B., Nusche, D., & Moorman, H. (2008). Improving school leadership, Volume 1:
Policy and practice. Paris, France: OECD. Retrieved from https://www.oecd.org/edu/school/44374889.pdf
Reeler, D. (2007). A three-fold theory of social change: And implication for practice, planning, monitoring and evaluation. Cape Town, South Africa: The Community Development Resource Association. Retrieved from
http://www.cdra.org.za/uploads/1/1/1/6/111664/threefold_theory_of_change_-_a nd_implications_for_pme_-_doug_reeler_of_the_cdra.pdf
Renger, R., & Titcomb, A. (2002). A three-step approach to teaching logic models. American Journal of Evaluation, 23(4), 493-503.
Robinson, V. M., Lloyd, C. A., & Rowe, K. J. (2008). The impact of leadership on student outcomes: An analysis of the differential effects of leadership types. Educational Administration Quarterly, 44(5), 635-674
Sabatier, P. A. (1999). Theories of the policy process. Boulder, CO: Westview. Sato, M. (2010). Challenges for schools: The construction of learning community (Q.
Q. Chong, Trans.). Shanghai, China: East China Normal University. (in Chinese) Sato, M. (2012). Learning revolution (Y. L. Huang & Q. Q. Chong, Trans.). Taipei,
Taiwan: Parenting. (in Chinese)
Savigny, D., & Adam, T. (Eds.) (2009). Systems thinking for health systems strengthening. Genève, Switzerland: World Health Organization.
Sergiovanni, T. J. (2001). The principalship: A reflective practice perspective (4th ed.). Needham, MA: Allyn and Bacon.
Spillane, J. P. (2005). Distributed leadership. The Educational Forum, 69(2), 143-150. Spillane, J. P., & Camburn, E. (2006). The practice of leading and managing: The
distribution of responsibility for leadership and management in the schoolhouse. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.
Sterman, J. D. (2000). Business dynamics: Systems thinking and modeling for a complex world. Boston, MA: McGraw-Hill.
United Way of America (1996). Measuring program outcomes: A practical approach. Arlington, VA: Author.