The Construction of an Evaluation Model for After-school Programs
1,2Amy H. I. Lee
, *1,3Chih-Neng Yang,
1Chun-Yu Lin
1
Ph.D. Program of Technology Management, Chung Hua University, Taiwan
2Department of Technology Management, Chung Hua University, Taiwan,
E-mail: amylee@chu.edu.tw
*3
Department of Marketing and Logistics Management, Ta Hwa Institute of Technology,
Taiwan,
E-mail:
aeycn@thit.edu.tw
doi:10.4156/aiss.vol3.issue4.18Abstract
During the past few decades, various types of after-school programs for children have been set up dramatically in Taiwan due to social structure changes. This research aims to construct an evaluation model for after-school programs, based on the viewpoints of theo ries and practices and incorporated both the service quality and the marketi ng strategy. After a tho rough literature review and in terview with experts, a n eva luation hierarchy, which con sists of t he fa ctors th at sho uld be co nsidered in evaluating after-school programs, is constructed. Experts are invited to fill out a questionnaire based on the hierarchy. Analytic hierarchy process (AHP) is used to calculate the relative importance of the factors. The d eveloped eva luation mod el sh all provide an eff ective and objective mechanism for evaluating after-school programs.
Keywords
: Service Quality, After-school Programs, Marketing Strategy, Analytic Hierarchy Process1. Introduction
The tremendous growth of after-school programs is due to three major societal concerns [10][11]. First, a dramatic shift in employment patterns has resulted in many children who are home alone and unattended for long periods of time after school [10][11]. The second societal concern is that children can learn under the right environment [10][11]. After-school programs, when operated effectively, can support students in their academic learning by providing them with structure, supervision, academic assistance, and the opportunity to learn study skills [21]. The third societal concern is the increasing trend of youth crime and youth victimization. After-school programs can occupy students’ after-After-school time so that they can avoid involving in dangerous or harmful behaviors and situations.
Research on after-school education has been increasing in recent years in various countries due societal changes. Cosden et al. [5] studied the influences of homework programs and after-school activities on after-school success in the U.S. and suggested to balance homework assistance with other aspects of the children’s home lives to promote positive developmental outcomes. Huang and Cho [6] also studied the after-school programs in the U.S. and recommended that after-school programs should provide students with ample time and resources to complete their homework and teaching students particular study skills. Turmo et al. [22] studied the primary school students in Norway and found that after-school care is of significance on science achievement since it could provide increased number of learning opportunities.
Due to the changes in Taiwan's social structure, women's labor force participation rate increases consistently, and after-school care has become a second home for many children. In addition, because of the changes of traditional family structure and lifestyle, coupled with declining birthrate phenomenon of the modern family, parents expect more from their children and are willing to increase the spending on children’s education. As a result, more and more parents decide to send their school-age children to after-school care and various classes.
In Taiwan, most after-school care services, such as after-school care centers, talent classes, and cram classes, are privately-owned. Basically, the course design is to meet the needs of the market and focuses on children's homework assistance and provide repeated practices on schoolwork. A variety of
talent courses are also provided to enhance children’s music, arts, mathematics, science, etc. Many parents believe that after-school education not only can take care of their children, but also can make good use of time to enrich the knowledge of all aspects of their children, in addition to the removal of their children from the dangers of unattended at home. The emergence of school cram classes in the market, however, also creates problems, including teaching quality, teachers’ qualifications, transportation security, environment safety and many other issues [12].
Most of the existing studies on school care services focused on the understanding of the after-school care education market. For example, Chiu [2], Huang [7], Lee [13], Lu [16] and Ma et al. [17] used different research methods to study after-school care services in Taiwan and to explore its extension of primary school curriculum. These studies found that after-school care was primarily aimed at academic counseling, and the major reasons parents let children attend after-school care institutions are to take advantage of time to finish homework and to strengthen various talents. Lee [14] studied the behavior of parents in choosing after-school care institutions. Hung [8], studied the parents of primary after-school classes and found that the quality of teaching service was positively related with the satisfaction of parents. Hung [9], in considering the declining birthrate and the trend of after-school care competition, studied whether the amount of parental involvement would affect the degree of satisfaction. The results showed that the degree of parental involvement had a significant impact on service quality and on parents’ satisfaction. Tsai [23] used a quality function deployment (QFD) model to examine the quality of primary after-school care services, and found that the important quality improvement items included service attitude, professional skill, communication ability, reaction ability, safe and comfortable environment, curriculum plans, advanced education training of teachers, administrative coordination and communication, administrative assistance in teaching and management, and propaganda and advertisement. Chen and Lu [1] studied the after-school time of adolescents in Taiwan and found that time spent on after-school academic-related activities had positive effects on the educational achievement of the students.
The existing studies on the after-school care in Taiwan are mostly about the understanding of the current conditions of the after-school care market, and the exposition of activities, services and service quality. There is a lack of study on the institutional evaluation criteria. Therefore, this study is to explore after-school care institutions through the viewpoint of service quality. The main purposes are to develop an effective and objective evaluation model for institution selection and to determine the weights and ranking of evaluation criteria.
The rest of this paper is organized as follows. Section 2 constructed an analytic hierarchy process (AHP) model for after-school program evaluation. Section 3 applies the model in a case study in Taiwan. Some conclusions and discussions are presented in the last section.
2. Methodology
This paper adopts the Delphi method and the analytic hierarchy process (AHP) approach to establish an evaluation model for after-school programs. The steps are as follows:
Step 1. Use the Delphi method to select the most important factors. A questionnaire with a five-point
Likert-type scale (where 1 means “strongly unimportant” and 5 means “strongly important”) is prepared, and experts are invited to fill out the questionnaire. The absolute value and the standard deviation of the difference between the mode and the mean are calculated for each factor to study the dispersion of the responses. When the absolute value is less than or equal to 1, the consistency of experts’ opinions is high; otherwise, the experts are asked to re-do the part of the questionnaire until a convergence is met. Select the factors with a mean that is greater than or equal to a threshold.
Step 2. Construct a hierarchy with dimensions and factors for evaluating after-school programs based
on the factors selected in step 1.
Step 3. Prepare a questionnaire based on the hierarchy. The AHP is adopted to develop relative
pairwise comparison weights of dimensions (factors), and a nine-point scale is used to express preferences between dimensions (factors) as equally, moderately, strongly, very strongly, or extremely preferred (with pairwise weights of 1, 3, 5, 7 and 9, respectively). Values of 2, 4, 6 and 8 are the intermediate values for the preference scale.
Step 4. Construct pairwise comparison matrices for each expert k. For example, if there are n factors,
denoted by X1, X2, X3, …, Xn, with relative weights, denoted by
w
1k,w
2k,w
3k, …,w
nk, respectively, the pairwise comparisons can be represented in the form of a matrix [4].
nnk k n k n nk k k nk k k ka
a
a
a
a
a
a
a
a
2 1 2 22 21 1 12 11 nk nk 2k nk 1k nk nk 2k 2k 2k 1k 2k nk 1k 2k 1k 1k 1kw
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
A
(1)Step 5. Compute the eigenvalue and eigenvector of each pairwise comparison matrix for each expert k.
k k
k
w
max
w
A
(2) where w is the eigenvector ofA
k, and
max is the largest eigenvalue ofA
k.Step 6. Check the consistency property of each matrix. The consistency index (CI) and consistency
ratio (CR) are defined as:
1 max n n CI
(3) RI CI CR
(4) where n is the number of items being compared in the matrix, and RI is random index [20]. If CR is greater than 0.1, it is an indication of inconsistent judgment, and the expert needs to revise the values in the pairwise comparison matrix.
Step 7. Aggregate multiple experts’ responses and form aggregated pairwise comparison matrices. By
employing geometric mean approach, an aggregated pairwise comparison matrix for K experts is: * * ij
a
(5) wherea
ij*
a
ij1
a
ij2
...
a
ijK
1/KStep 8. Compute the eigenvalue and eigenvector of each aggregated pairwise comparison matrix and
check the consistency property of each matrix.
Step 9. Calculate the synthesized priorities of the factors. The overall priority of a factor can be
obtained by multiplying the factor priority with the priority of its upper-level dimension.
3. Case study
Service quality management is currently a subject with considerable attention in the business community [15][18]. The concept of service quality can also be adopted in education to provide effective management and to enhance the service quality of education [3]. The marketing function of after-school care is also important to the successful fulfillment of educational mission and the enhancement of public and market satisfaction [19]. Therefore, the hierarchy for evaluating
after-school programs is divided into two dimensions: service quality and marketing. Based on literature review and interview with experts, factors for achieving service quality are listed. A group of nine experts in the field of after-school programs are invited to determine the importance of these factors through the Delphi method. Five experts belonged to the parents group, and four represented the operators group. After the second-round questionnaire, stability in the results is achieved. The absolute value of the difference between the mode and the mean is less than 1, and the mean decreases from 0.58 in the first run to 0.51 in the second run. The factors with an importance greater than 4 are selected for further analysis. The five factors under the service quality dimensions are: fee and location, facility and hygiene, teacher qualification and counseling, administration and services, and communication and attentiveness. For the marketing dimension, five P’s., i.e. product, price, people, place and promotions, are the factors. The hierarchy is depicted in Figure 1.
A questionnaire based on the hierarchy is filled out by the experts, pairwise comparison matrices for each expert are prepared. For example, the pairwise comparison matrix of the factors under the service quality dimension for the first expert is as shown in Table 1.
Figure 1. The hierarchy for evaluating after-school programs Table 1. Comparison matrix for the factors under service quality by expert 1.
Fee and location (FS1) Facility and hygiene (FS2) Teacher qualification and counseling (FS3) Administration and services (FS4) Communication and attentiveness (FS5)
Fee and location
(FS1) 1 2 1/2 2 2 Facility and hygiene (FS2) 1/2 1 1 5 2 Teacher qualification and counseling (FS3) 2 1 1 6 2 Administration and services (FS4) 1/2 1/5 1/6 1 1/2 Communication and attentiveness (FS5) 1/2 1/2 1/2 2 1
An eigenvector and an eigenvalue are calculated using the eigenvalue method by Equation (2). S1 S2 S3 S4 S5 F 0.239 F 0.244 F 0.323 F 0.067 F 0.127 w and
max= 5.3292The eigenvector shows the priority of the five factors assessed by the first expert. To check the consistency of this combination of values in the matrix,
max is substituted into Equation (3) to obtain CI, and CR is calculated by Equation (4).max 5.3292 5 0.0823 1 5 1 n CI n , 0.0823 0.0735 RI 1.12 CI CR
Since CR is less than 0.1, the comparison matrix is consistent.
Table 2 shows the weights of dimensions and factors and the consistency test results generated for each individual expert (E1 to E9). After all the comparison matrices have passed the consistency test, the opinions of the experts are aggregated by a geometric mean method. For example, the pairwise comparison results between fee and location (FS1) and facility and hygiene (FS2) from the nine experts are (2:1), (1:1), (1:4), (7:1), (4:1), (2:1), (4:1), (1:3) and (1:1). The opinions are aggregated by the geometric mean method, and the synthesized rating is (2 1 0.25 7 4 2 4 0.33 1) 1/91.495. A
combined matrix for the five factors under service quality dimension is calculated:
Fs1 Fs2 Fs3 Fs4 Fs5 Fs1 1 1.495 0.558 1.501 0.573 Fs2 0.669 1 0.524 1.107 0.381 s A Fs3 1.790 1.908 1 2.273 0.740 Fs4 0.666 0.903 0.440 1 0.312 Fs5 1.744 2.621 1.351 3.198 1
Table 2. Results from individual experts.
The eigenvector, eigenvalue, CI and CR of the synthesized matrix are: 1 2 3 4 5 F 0.172 F 0.125 F 0.260 F 0.111 F 0.332 S S s S S S w ,
max= 5.0148 , CI 0.0037, CR 0.0033After all calculations, the final priorities and rankings of the dimensions and factors are shown in Table 3. Under the service qual ity dimension, the most important factor is communication and attentiveness, with a local priority of 0.332, followed by teacher qualification and counseling (0.26), fee and location (0.172), facility and hygiene (0.125), and administration and services (0.111). As a result, communication and attentiveness is the most important issue in selecting after-class institutions. Teachers should maintain contact with parents to inform students’ learning conditions, as well as focus on their qualification and counseling. Institutions need to provide services with efficiency and help resolve the problems of students and parents promptly.
Weights by experts
Dimensions/Factors E1 E2 E3 E4 E5 E6 E7 E8 E9
Dimensions Service quality 0.870 0.750 0.833 0.833 0.875 0.750 0.833 0.750 0.857
Marketing strategy 0.125 0.250 0.167 0.167 0.125 0.250 0.167 0.250 0.143
Factors
Service quality
Fee and location (FS1) 0.239 0.210 0.033 0.389 0.208 0.075 0.431 0.046 0.086
Facility and hygiene
(FS2) 0.244 0.203 0.079 0.043 0.069 0.054 0.141 0.157 0.065 Teacher qualification and counseling (FS3) 0.323 0.064 0.452 0.092 0.385 0.200 0.130 0.224 0.389 Administration and services (FS4) 0.067 0.037 0.223 0.043 0.039 0.066 0.234 0.377 0.059 Communication and attentiveness (FS5) 0.127 0.485 0.212 0.433 0.299 0.605 0.065 0.197 0.401 Marketing strategy Product (FM1) 0.052 0.248 0.116 0.033 0.446 0.033 0.430 0.070 0.047 Price (FM2) 0.141 0.079 0.532 0.356 0.100 0.107 0.218 0.482 0.217 People (FM3) 0.135 0.173 0.036 0.102 0.050 0.382 0.130 0.159 0.140 Place (FM4) 0.062 0.051 0.083 0.149 0.352 0.068 0.092 0.195 0.088 Promotion (FM5) 0.610 0.450 0.233 0.360 0.052 0.410 0.130 0.093 0.508 Consistency test Service quality λmax 5.329 5.426 5.305 5.376 5.203 5.322 5.356 5.407 5.345 CI 0.082 0.106 0.076 0.094 0.051 0.081 0.089 0.102 0.086 RI 1.120 1.120 1.120 1.120 1.120 1.120 1.120 1.120 1.120 CR 0.073 0.095 0.068 0.084 0.045 0.072 0.080 0.091 0.077 Marketing strategy λmax 5.417 5.439 5.391 5.328 5.374 5.372 5.430 5.335 5.363 CI 0.104 0.110 0.098 0.082 0.094 0.093 0.108 0.084 0.091 RI 1.120 1.120 1.120 1.120 1.120 1.120 1.120 1.120 1.120 CR 0.093 0.098 0.087 0.073 0.084 0.083 0.096 0.075 0.081
Table 3. Synthesized results of the dimensions and factors.
Dimensions Factors Local
priorities Local ranking Integrated priorities Integrated ranking Service quality (0.823)
Fee and location (FS1) 0.172 3 0.1413 3
Facility and hygiene (FS2) 0.125 4 0.1027 4
Teacher qualification and counseling (FS3) 0.260 2 0.2139 2
Administration and services (FS4) 0.111 5 0.0913 5
Communication and attentiveness (FS5) 0.332 1 0.2733 1
Marketing strategy (0.177) Product (FM1) 0.129 5 0.0229 10 Price (FM2) 0.259 2 0.0459 7 People (FM3) 0.152 3 0.0269 8 Place (FM4) 0.137 4 0.0243 9 Promotion (FM5) 0.324 1 0.0574 6
Under the marketing strategy dimension, promotion (0.324) ranks the first, followed by price (0.259), people (0.152), place (0.137), and product (0.129). The result shows that promotional campaign of educational products to communicate to and to persuade potential customers, as well as the price of products and services, is particularly important.
Because the service quality dimension has a very high priority of 0.823, the top five factors in the hierarchy all belong to this dimension, and they are, in sequence of importance, communication and attentiveness (0.273), teacher qu alification and counseling (0.214), fee and location (0.142), facility and hygiene (0.103), and administration and services (0.091).
4. Conclusions
This paper constructs a model for evaluating the performance of after-school institutions in Taiwan by adopting the AHP. The results show that factors from the service qu ality dimension are highly emphasized. In the educational services market, how to provide effective services and incorporate quality into marketing strategy enable institutions to offer superior service quality. The use of promotional activities to sell their services and to increase the interests of potential customers is also important. Government should, through evaluation and reward systems, help after-school care institutions to set up good-quality childcare and to create a happy learning environment. The management and operation of an institution, in addition to consider the parents’ expectations and the market trend, needs to focus more on the needs of students in order to provide students the best after-school care services. Finally, the developed evaluation model can be used or tailored by interested parties as an effective and objective mechanism for evaluating after-school programs. The proposed model can be tailored in the aspects studied or the factors being considered to suit the different after-school needs and after-after-school programs in different countries. In addition, children of different age categories may require different after-school cares; therefore, different after-school programs can be categorized, and evaluating hierarchies can be revised accordingly to meet the needs of different age groups.
5. References
[1] Chen, S.Y., Lu, L., “After-school Time Use in Taiwan: Effects on Educational Achievement and Well-being”, Adolescence, Vol. 44, pp. 891-909, 2009.
[2] Chiu, D.H., The Study of Consumer Behavior for Day Care and Tutorial Center, Unpublished master dissertation, Graduate school of business administration, Dayeh University, Taiwan, 2000. [3] Chuang, C.-F., Cheng, C.-J., “A Study of Institutional Repository Service Quality and Users’
Loyalty to College Libraries in Taiwan: The Mediating & Moderating Effects”, Journal of Convergence Information Technology, Vol. 5, No. 8, pp. 89-99, 2010.
[4] Chung, S.H., Lee, A.H.I., and Pearn, W.L., “Product Mix Optimization for Semiconductor Manufacturing Based on AHP and ANP Analysis”, International Journal of Advanced Manufacturing Technology, Vol. 25, pp. 1144-1156, 2005.
[5] Cosden, M., Morrison, G., Gutierrez, L., Brown, M., “The Effects of Homework Programs and After-school Activities on School Success”, Theory into Practice, Vol. 43, No. 3, pp. 220-226, 2004
[6] Huang, D., Cho, J., “Academic Enrichment in High-functioning Homework Afterschool Programs”, Journal of Research in Childhood Education, Vol. 23, No. 3, pp. 382-392, 2009. [7] Huang, I.C., “After-school Child Care Arrangements in Tainan: Needs and Factors Associated
with Parents’ Decisions”, Journal of National Tainan Teachers College, Vol. 33, pp. 233-262, 2000.
[8] Hung, C.E., A Study on Service Quality and Satisfactory Factor of the After-school Accomplishments Classes, Unpublished master dissertation, Graduate school of business administration, Dayeh University, Taiwan, 2008.
[9] Hung, M.Y., A Study of the Relationship among Service Quality, Involvement and Satisfaction - Based on Private and Public Kindergarten in Taichung Country, Unpublished master dissertation, Graduate school of business administration, Dayeh University, Taiwan, 2009.
[10] Kugler, M.R. “After-school Programs are Making a Difference”, National Association of Secondary School Principals (NASSP) Bulletin, Vol. 85, pp. 3-11, 2001a.
[11] Kugler, M.R., “The Why and How of After-school Programs”, National Association of Secondary School Principals (NASSP) Bulletin, Vol. 85, pp. 44-50, 2001b.
[12] Lee, H.M., “America's After-school Program and its Implication on School-age Child Care in Taiwan”, Kaohsiung Teachers College, Vol. 13, pp. 235-256, 2002.
[13] Lee, H.M., “The Continuity and Integration: From Elementary School Curriculum to After-school Care Program”, Journal of Child Care, Vol. 1, pp. 19-38, 2003.
[14] Lee, Y.H., The Consumer Behavioral Study of Selecting After-school Tutorial Institutes by Parents of Elementary Students in Taipei City, Unpublished master dissertation, Department of Society Education, Taipei Municipal University of Education, 2007.
[15] Liu, R., Bei, H., “Research about the University Students Incubator Service Quality based on the TQM Approach”, Advances in Information Sciences and Service Sciences, Vol. 3, No. 1, pp. 123-131, 2011.
[16] Lu, C.W., The Research of Student’s Participation of After School Studying in Hsinchu Local Elementary School, Unpublished master dissertation, Department of Education, National Hsinchu University of Education, 2007.
[17] Ma, T.L., Cheng, P.W., Cheng, S.H., Su, Y.P., and Hseih, H.R., “After-school Child Care Setting: A Case Study”, The Journal of Health Sciences, Vol. 2, No. 1, pp. 1-11, 2000.
[18] Pan, W.-T., Chen, P.-W., “A Study on the Logistic Service Satisfaction for Internet Marketing Enterprise Using Data Mining Technology”, Advances in Information Sciences and Service Sciences, Vol. 3, No. 2, pp. 114-120, 2011.
[19] Salcedo, M. H., Vilariño, D. A., López, F. L., Somodevilla, J. G., “Clearing Policy for an Specialist Agent in Marketing Strategies”, Journal of Convergence Information Technology, Vol. 5, No. 7, pp. 78-83, 2010.
[20] Satty, T.L., The Analytical Hierarchy Process, McGraw-Hill: New York, 1980.
[21] Scott-Little, C., Hamann, M.S., and Jurs, S.G., “Evaluations of After-school Programs: A Meta-evaluation of Methodologies and Narrative Synthesis of Findings”, American Journal of Evaluation, Vol. 23, No. 4, pp. 387-419, 2002.
[22] Turmo, A., Guttersrud, O., Elstad, E., Olsen, R.V., “The Impact of Attending After-school Care Schemes on Science Achievement in Primary School: A Norwegian Study”, International Journal of Educational Research, Vol. 48, pp. 331-341, 2009.
[23] Tsai, M.L., A Study of Quality on the Service of After-school Care at Elementary Schools by QFD in Taichung County, Unpublished master dissertation, Graduate School of Business Administration, Dayeh University, Taiwan, 2009.
