Developing the e-Delphi system: a web-based forecasting tool for educational research

全文

(1)10_Chou 14/03/2002 1:11 pm Page 233. British Journal of Educational Technology. Vol 33 No 2 2002. 233–236. Colloquium. Developing the e-Delphi system: a web-based forecasting tool for educational research Chien Chou Address for correspondence: Institute of Education, National Chiao Tung University, Hsinchu, Taiwan 30010. Tel: +886-3-5731808; Fax: +886-3-5738083; email: cchou@cc.nctu.edu.tw. Introduction The Delphi technique has become a widely used tool in a variety of disciplines (Rowe and Wright, 1999) for measuring and aiding forecasting and decision making since 1960. Dalkey and Helmer (1963) claim that this technique produces a reliable consensus among an expert group by a series of intensive questionnaires interspersed with controlled opinion feedback. Typically, a Delphi study is conducted in a number of rounds. In the first round, a questionnaire is created, sent to panel members to complete and return, and the responses are analyzed. In the next round, a new questionnaire is developed from the previous responses, and then sent to panel members. The aim of the series of questionnaires is to achieve a consensus of opinion by allowing members to re-consider and re-rate their opinions regarding the items in the questionnaire. The Delphi technique undoubtedly has both advantages and disadvantages; one of the major disadvantages is its potential for sloppy execution (Gupta and Clarke, 1996). That is, Delphi studies relying on traditional mailing systems require relatively long periods of time to construct questionnaires and collect panel members’ opinions. In addition, during the course of execution, some members may not complete or return the questionnaires for each round; therefore, valuable member contributions may be lost. In order to resolve these problems, an e-Delphi system was developed, in which all questionnaire construction and communication with panel members was accomplished using the Web. System users and requirements There are two types of users identified by the e-Delphi system: the project leader, who conducts the study, and panel members, who answer questionnaires. The e-Delphi system is primarily for project leaders to conduct Delphi studies more efficiently and effectively, but it also benefits panel members, enabling them to link directly to the system and enter their opinions. The basic requirements for the e-Delphi system are as follows: 1. Provide a friendly interface that allows the project leader to develop and send questionnaires to panel members. © British Educational Communications and Technology Agency, 2002. Published by Blackwell Publishers Ltd, 108 Cowley Road, Oxford, OX4 1JF, UK and 350 Main Street, Malden, MA 02148, USA..

(2) 10_Chou 14/03/2002 1:11 pm Page 234. 234. 2. 3. 4. 5. 6.. British Journal of Educational Technology. Vol 33 No 2 2002. Provide a friendly interface that allows panel members to input data. Perform calculations on panel members’ input entries. Prepare individual questionnaires with multimedia presentation. Help the project leader determine the stability of each item of the questionnaire. Allow the project leader to monitor the execution of the study and to communicate with panel members easily.. The resulting e-Delphi system was built on the Web, using Active Server Page (ASP) techniques. For readability, technical details are omitted and only the functions and some user-interfaces are introduced below. System functions and interface The system interface consists of three areas shown on the screen. The first consists of information on the current study, including its name, current round number, and number of panel members who have completed the questionnaire (if round number is two or higher). The vertical area at the left of the screen lists all Delphi function buttons. Project leaders may select any function and work within the largest area beside the function lists on the screen. There are seven major function buttons, briefly described below: 1. Panel members. Project leaders must first establish panel members for any new Delphi study. This function can create accounts for new members, as well as view and delete accounts from an existing member list. 2. Questions. The next step is to develop the study questionnaires. This function can create new questions, and view and delete existing questions. 3. Scale. After developing the questionnaire, project leaders should determine the scale and scale indicators for the questionnaire. For example, after project leaders enter “5” on the scale, the system will ask them to enter the verbal indicators for 1, 2, 3, 4, and 5; project leaders can enter Very Disagree, Disagree, Neutral, Agree, or Very Agree for each indicator. 4. Email. This function allows project leaders to write cover letters and follow-up letters to be sent to panel members. The system will automatically add the Web address of the Delphi study, panel members’ account numbers, and the password to this and subsequent letters. When panel members receive email letters, they can click the Web address included in the letter, access the system, and enter their account numbers and password in order to access the questionnaire. 5. Questionnaire. This function allows project leaders to overview and modify each questionnaire, including the introduction, each question and its order, scale and indicator. Figure 1 shows a second round questionnaire for panel members. 6. Round. This function allows project leaders to (a) send a follow-up email letter to members who did not complete the current-round questionnaire, and (b) send the next-round cover letters and questionnaires to panel members. 7. Statistics. This function allows project leaders to calculate panel members’ inputs for each round. The system will calculate the number of panel members who answered any given question, as well as the mean, standard deviation, mode, quartile deviation (this helps project leaders determine the consensus of each question), and net © British Educational Communications and Technology Agency, 2002..

(3) 10_Chou 14/03/2002 1:11 pm Page 235. Colloquium. 235. Figure 1: The e-Delphi system showing the questionnaire for a panel member. person-changes (this helps determine the stability of each question). This function automatically indicates each question’s degree of consensus and stability by different colors. Evaluating the e-Delphi system In order to evaluate the usability and interface design of the e-Delphi system, a study entitled “Information and Communication Technology Educator” (ICT Educator) was conducted, in which 15 panel experts participated. Panel members, all of whom specialize in information and communication technology, experienced no difficulty in accessing the system or completing and submitting their questionnaires for each round. After completing the third round questionnaire, members were required to answer an additional questionnaire on the use of the system. Their answers indicated that the use of email and the Web in this Delphi study made it easier for them to participate and complete their tasks than other Delphi studies in which they had participated. In addition, since they are used to typing, rather than handwriting, they were more willing to enter their opinions to open-end questions. They recommended that if a Delphi study focuses on network technology, and all panel members use Internet applications regularly, then it is best to employ on-line Delphi methods such as the one described in this paper. © British Educational Communications and Technology Agency, 2002..

(4) 10_Chou 14/03/2002 1:11 pm Page 236. 236. British Journal of Educational Technology. Vol 33 No 2 2002. From the viewpoint of a Delphi study project leader, the major advantage of this system is that it saves significant execution time, including developing and sending individual questionnaires, sending follow-up letters and accompanying questionnaires, performing statistical functions, and determining the consensus and stability of each question. The execution of the entire ICT Educator study included three rounds of questionnaire distributions, two follow-up letters, and data calculation; the total execution time was four weeks, compared with the average six to 12 months in a traditional Delphi study. In addition, this system retains the merits of the traditional Delphi system in which panel members are anonymous to each other, while providing an easy channel for members to communicate with the project leader. For example, two panel members sent emails to the project leader regarding their schedules and available time for answering questionnaires. Final remarks The e-Delphi system described in this paper is an innovative approach to an existing forecasting method; this system will continue to be revised for a higher level of “userfriendliness”. The e-Delphi system is a less labor-intensive system than the traditional method, and is not paper-reliant; it retains the essence of traditional methods, but speeds up the execution process. This and other developing technologies will continue to offer Delphi project leaders and panel experts the benefit of Web-based research tools. Acknowledgement This work was supported by the National Science Council in Taiwan under Project NSC892520-S-009-005. References Dalkey N C and Helmer O (1963) An experimental application of the Delphi method to the use of experts Management Science 9 458–467. Gupta U G and Clarke R E (1996) Theory and applications of the Delphi technique: A bibliography (1975–1994) Technological Forecasting and Social Change 53 (2) 185–211. Rowe G and Wright G (1999) The Delphi technique as a forecasting tool: Issues and analysis International Journal of Forecasting 15 (4) 353–375.. © British Educational Communications and Technology Agency, 2002..

(5)