The design and the
The design and the
formative
formative
evaluation of a
evaluation of a
Web-based course
Web-based course
for simulation
for simulation
experiences
experiences
(Computers &
(Computers &
Education, 2006 )
Education, 2006 )
Tao, Y.-H.
Tao, Y.-H.
Guo, S. and Lu, Y.
Guo, S. and Lu, Y.
亞太工商管理研究所：資訊管理個案
Background-Importance of Computer Simulation
Importance of Computer Simulation
• An important skill
– a seasoned professional in operation research/management scienc
e (OR/MS)
(Dyer, Bean, Dewald, Gepfert, & Odoni, 1993)– a critical OR tool in the industry
(Nelson, Bradbard, Ledbetter, & Cox, 1987)• A typical simulation process
(Taylor & Hurrion, 1988a, 1988b)– formulation
– programming
– verification
– validation
– experiment design and analysis
– recommendations
Background
Background
Learning Time
Learning Time
• Shannon (1985) estimated that one needs
to receive
‘
‘
at least 720 h of formal class
instruction plus another 1440 h of outside
study’’
to acquire the basic simulation
tools
•
• This does not include the
extra effort
required to gain
real-world experience
in
Background
Background
Classroom Learning
Classroom Learning
• Students usually take
only
one
simulation
course
• Time constraints : often limited to
simulation concepts
,
modeling
and
programming
(Lu, 2000; Rohrer & Banks, 1998)• Simulation
design and analysis
– is
simplified
and receives a lower priority
– most students consider it a much more
difficult topic
• However, a successful simulation application depends heavily
on the
effectiveness of simulation statistical analysis
• Thus,
efficient learning
of
effective simulation statistical
analysis
merits more research
Background-CBT
CBT
• Computer-based training
(CBT) has been widely applied in learning
after the burgeoning
popularity of personal computer
in the
eighties
• According to the
survey
of Whitehouse and Pellegrin(1995),
utilizing personal
computer and software
to raise the knowledge of
students can
save up to70% of training time
• Recent popularity of
Internet
has brought more benefits into
traditional CBT learning
– 24 -7 availability
– better interactions between students instructors
– virtual classroom space
• An
asynchronous
web-based CBT system can serve as an
after-hour teaching assistant
to traditional classroom learning
Objective
Objective
• The
tacit knowledge
of
experiential
simulation analysis is
not
included in major
English
textbooks
, nor are these
simulation-related CBT systems currently
available to the public
to our knowledge
• Objective
– to explore the design of a web-based CBT
system that can assist the beginners to study
effective simulation analysis experience in an
efficient way
Literature –
Literature –
Content of experiential simulation
statistical analysis
• Teaching
both
the
knowledge and the
experiences
of simulation statistical analysis are required to help
novice acquire effective problem-solving skills (Tao,
1999)
• Experience
is a broad term and the possible
categories of simulation experience include
– high level
simulation-analysis framework
– practical simulation-analysis
models
– collections of simulation
tactics
– case studies
from the software companies, consulting firms
or conference proceedings
– useful simulation-analysis
processes in the literature
Only tactics
could not be
found in the
references
Literature –
Literature –
Content of experiential simulation
statistical analysis
• The
practical experience
of simulation statistical analysis
– received less attention in the literature
– Mellichamp and Park (1989), Ramachandran, Kimbler, and Naadimuthu (198 8), Taylor and Hurrion (1988a, 1988b), and Tao and Nelson (1997) were t he only few who had investigated simulation analysis
• On the
applications
, Goldsman, Nelson, and Schmeiser (1991) describ
ed their simulation
problem-solving processes
that solved an airlin
e reservation system problem using
three different approaches
, all
reaching the same conclusion
• Tao and Guo (2000) proposed a
mental model
for simulation statistic
al analysis, which included two cognitive guidelines, two design in
dicators and a heuristic problem-solving process
• They
cannot
be easily
organized for or shared
during the classroom
settings
Literature –
Literature –
Concept of interactive system design
• Norman (1986) proposed a seven-stage model of interaction
development
– The evaluation measures the final interpretation against the initial goal – Goal setting is the most important foundation for any interactive
system development.
– Newman and Lamming (1995) pointed out that supporting activity, form of solution, user, and level of support are important elements of problem definition
• Newman and Lamming (1995) also proposed a
method for
conceptual design
– identifying the form of mental model, – hiding the mental model,
– encouraging the acquisition of this mental model by providing a suitable system image
– analyzing the design by means of detailed cognitive walkthrough – applications of design heuristic and guidelines
Literature –
Literature –
Concept of interactive system design 2
• Guidelines:
– Used when not familiar with conceptual design (Shneiderman, 1992)
– Knowing, selecting, and applying design guidelines are very important a s design guidelines may have supplemental, conflicting, or overlapping effects.
– Too many design guidelines, the marginal effect shows
– Good references for design guidelines (Shneiderman, 1992; Nielsen, 2000 ; Johnson, 2000)
• Evaluation is to support the design,
formative evaluation
is prefer
red (Preece et al., 1996).
Literature –
Literature –
Characteristics of Web-based CBT
• Criticism
– schools for rushing into distance learning by offering inexpensive ‘‘ plug and pay’’ courses, and claimed that online learning cannot repla ce life on a real campus (Ubell, (2000)
– a rebottling or a new brew (Wiesner, 2000)
– limitations and pitfalls in web-based education as a major field of res earch and a hot area of applications (Ausserhofer, 1999)
• 80% of asynchronous learning and 20% of the synchronous learning
fo
r better learning experience. The driving forces behind their 80/20
model were three learning models, including Gagne’s nine instructi
onal events (Danchak, Jennings, Johnson, and Scalzo, 1999)
Literature –
Literature –
Characteristics of Web-based CBT2
• Zu and Chang (1998) organized the
taxonomy of learning styles
for C
BT,
– constructive learning – situated learning – case-based learning – apprenticeship learning – project-based learning,– story-based learning and collaborative learning
• Examples:
– a dynamic guidance using quiz functions for web education, which uses f unctions of on-line quiz, student profile, dynamic student grouping, an d user interface agent to provide interactive guidance function during learning (Okada, Tarumi, & Kambayashi, 2000)
– a hybrid course delivery model emphasizing the involved activities, in which hands-on experience and mini-lecture are two major components (Si ddiqui & Zubairi, 2000).
Research design and scope
Problem definition
• ‘‘to design an efficient yet effective web-based CBT course to assist the simulation novice learn simulation tactics in conjunction with the traditional classroom learning’’
• The design of this web-based CBT course focused on the experiential part
of simulation statistical analysis and the learning-enhancement interactions.
• The target users were college students with basic statistical training and had learned or were in the process of learning simulation
• effectiveness was accomplished by embedding simulation statistical expertise in the CBT course
• efficiency was achieved by combining the learning of basic knowledge of simulation analysis in-class and the learning of simulation statistical
Research design and scope
User profile
• To meet
Usability goals
, we first
profiled the
targeted user
group based on our observations:
– the user was
computer literate
but might not be at the
expert level
– had
web experience
but minimum contacts with web
learning systems
– had
knowledge
of introductory
probability and statistics
but might not be familiar with statistical software
– liked to
practice during learning
– cared more about
quality but not quantity of learning
– wished to
gain problem-solving skill
in a short period of
time
– was interested in
learning
simulation statistical analysis in
a
less stressful environment
Research design and scope
Model of experiential simulation
analysis
• A three-level model
– contextual
problem-solving guidelines
– problem-solving
process
– learning-unit design
• It could easily provide a
customized
learning experience for n
ovice through a mixture of what Zu and Chang (1998) called
con
structive learning
,
situated learning and case-based learning
.
• In particular, the
situated learning was incorporated into the
case-oriented learning
as focused scenarios
finest tactics the expert applied to solving a probl em, and were embedded in t he problem-solving process
specifically designed for the novice
the segmentation of the lea rning contents, which inclu ded both the guidelines and
Research design and scope
Interaction design guidelines
• The challenge was to
select, apply, and confirm a set of design
guidelines
that aggregated to an acceptable level of
usability
goal
• Based on our previous experience and study (Tao, 2000),
trial-a
nd-error
approach and hoped to achieve
80% of intended goals
wi
th a minimum set of guidelines.
• This research applied
– instructional design guidelines proposed by Gagne’ et al. (1988)
– interface design guidelines proposed by Zu and Chang (1998), Dix, Finlay, Abowd, and Beale (1998), Marcus (1992) and Shneiderman (19 92)
– self-inducted guidelines from experiences and observations over th e Internet
Research design and scope
Evaluation design
(a) User’s testing evaluation.
– informal evaluation on the initial design of the teaching system and the interface,
– users provide feedback as much as possible and as early as possible.
– observation and audio recording
– retrospective interviews and verbal protocol analyses (b) Expert’s constructive and executive evaluation
– internal validation checks of the learning material were performed by two simulation-analysis professors and a graduate student.
– evaluate the contextual design, instructional style and interface design from external experts perspective by three experts heuristic evaluation
– retrospective interviews
(c) User’s summative evaluation
– understand the usability and learning effects of the final prototype system
– beginners who had one year of probability and statistics, and one simulation class – The subjects were divided into
• the control group who studied the written material
• the experiment group who operated the CBT system
Design of prototype
Contextual problem-solving guidelines
• based on
– framework and the concept of sequential experim
ent from Tao and Nelson(1997)
– expert experiences from Goldsman et al. (1991)
and Kleijnen (1987)
• Not comprehensive but adequate
to validate
the experts problem-solving processes and t
o derive a simple and useful flow to assist
the learning of an online simulation analys
is system
Design of
prototype
Problem-solving
process
Design of prototype
Contextual problem-solving process
Step 1 judges the type of system and may apply guideline 2-1.
Step 2 decides on a project to start with and may apply guidelines 2-3 and 2-4.
Steps 3 and 4 determine whether or not to apply variance reduction technique (VRT), such as Common Random Number (CRN). Step 5 simulates minimum data for planning next design and may apply guidelines 1-1 and 1-2.
Step 6 enters the pilot run procedure that starts with experimental design (guidelines 3-1, 3-2, 3-3, and 3-4), followed by simulation execution and output data analysis (guidelines 2-2, 4-2, 6-3, and 6-4). At the end, guideline 4-1 may be used to judge whether remaining execution time is enough.
Step 7 compares alternative systems and draws conclusions based on guidelines 5-2, 5-3, and 6-2.
The above process is based on the multiple system comparison.
For single system evaluation, the process still starts with Step 1, but skips Step 7.
To incrementally design and evaluate simulation problems. It is intended to be a simplified expert model for beginners and emphasizes learning problem solving while building an appropriate mental model
Design of prototype
Learning-unit design
• The learning environment includes five major instruction units
– Learning guidance provides a roadmap for three different skill levels of users.
– Introduction to simulation briefly introduces simulation characteristics, applicable domains, and its pros and cons.
– Simulation theory presents fundamental simulation knowledge that includes theory and techniques in input/output data analysis.
– Design strategy includes both the problem-solving guidelines and problem-solving process as described in Sections 4.1 and 4.2.
– Case-oriented learning assists the users to learn from solving a problem using embedded guidelines and expert process. This is the major
Design of prototype
Interaction design guidelines and
demonstrations
The left side of screen shows the five teaching units (circle A). The discussion of the r ight side of the screen is divided into eight parts as follows.
(1) The upper screen shows the path of the learning contents (circle B), ‘‘case-oriente d learning > single system instance > steady state > start simulation > determining s ample size’’, which not only indicates current position, but also provides hyperlin ks to previous screens.
(2) Clicking on the palm-like image near the right edge of the screen (circle C), a windo w pops up that explains the purpose of current page. This applies the guideline for p roviding instant online help on an as – needed basis. The first two parts, (1) and (2), apply guidelines of avoiding lost, reducing short – term memory load, stimulati ng recalls of prior learning and providing convenience reversal browsing.
(3) The center of the main screen displays the dialog between the user and the system. As indicated by the path in (1) above, the current step is determining the initial sampl e size at the very beginning of the simulation process. The system prompts the user t o decide the sample size and simulation time. The user can click on the hyper-linked phrases ‘‘sample size’’ or ‘‘simulation time’’ (circle D) for explanations be fore making decisions. The applicable guidelines are providing online help for critic al terminology and hiding unnecessary information.
Design of prototype
Interaction design guidelines and
demonstrations
(4) The prompted system message in (3) is shown in the black color with blue underline
representing hyperlinks. The prompted question asking for sample size is shown in the red color with five green buttons for selection. The guidelines applied here are using up to seven colors for segmenting different purposes of information and providing stimulus materials with distinct features.
(5) The green buttons (circle E) provide alternative answers to the current question. If an inappropriate answer is chosen, the system brings up an explanation window instead of proceeding to the next step. After the user exits the pop-up window, the system retains the same question until appropriate answers are selected. The purpose is to provide situated learning environment, emphasize constructing knowledge by user, provide information feedback and provide learning guidance.
(6) The question in the dialog actually represents one of the difficult decisions of
simulation statistical analysis for beginners. Experiential guideline 1-1 is hidden during the process described in (5).
(7) A little flow diagram (circle F) appears on the right side of the screen, just above the blue palm-like image. Clicking on the image will display a full screen of the problem-solving flow diagram as seen in Fig. 2. The shaded blue area (arrow G) within the flow diagram represents the current step of the problem-solving process. The guidelines applied are providing recall of prior learning and providing learning guidance.
Design of prototype
Interaction design guidelines and
demonstrations
Fig. 4 displays a similar screen as shown in Fig. 3 except without the palm-like image and the red color question. The purpose is to be consistent and providing only necessary
information or functions.
(1) The bottom of the screen shows a pink question mark (arrow A). Clicking the image will pop up the explanation (of initial bias detection) for that option. It is a similar function like the palm-like image in Fig. 3 but only applicable when the user needs to choose the next step. The applicable guideline is also providing instance online help on a needed basis. (2) Either the palm-like image or the pink question mark provides optional online help.
However, the user may not know their usage at the first glance. Therefore, the system provides balloon help (arrow B as shown in the bottom edge of the screen) that offers a brief explanation when the cursor is near the image. The purpose is to provide interactive proactive clues.
(3) The screen layouts in Figs. 3 and 4 address the concise guideline by adapting the following features: page-long content, short paragraphs, wide spacing between paragraphs and sentences, hiding unnecessary information with hyperlink, color segmentation,
hierarchical information with selection buttons, and primary information in the middle with bright background color.
(4) In order to retain users learning interests for long hours of complicated and difficult domain knowledge and experience, the system embeds cartoon or scenic images in the learning activities as well as background music to entertain the users.
Evaluation and conclusion
Testing evaluation
• This stage evaluated the early prototyping system through observation and interviewing of users, who each performed a 50-min experiment as shown in Table 1. Since this was a developmental stage evaluation, the goal was to validate whether the design concept met the users needs
• Table 2 summarizes the evaluation results Two subjects participated in the experiment.
– The first subject, who had a simulation course, focused more on the content and had responded with 12 questions
– The second subject, who never had a simulation class, was interested on the interface and had more questions about the color design and the layout.
• On the content
– the first subject clearly knew what to expect and thought that some descriptions of simulation concepts were too tedious and the wording should be more precise
– the second subject was obviously troubled by many unfamiliar terms
• On the color design and the layout
– the second subject endorsed the use of buttons, lists, and tables – the first subjects main concern was the colors of the text.
Evaluation and conclusion
Constructive and executive evaluation
• This stage of evaluation was conducted after the framework
of the prototype system was drafted
with little details
• The goal was to understand the
learning material
and the
appropriateness of the teaching and interface design
• Four domain experts
participated in the evaluation of the
prototype framework. Each expert explored the system
differently with
an average time of 80 min
.
• Opinions were collected
during the operations
and
Evaluation and conclusion
Summative evaluation
• We conducted a
user evaluation to 30 university students
who had a 3-credit-hour simulation introductory course
• The students were divided into
• a
control group
studying a written material
• an
experiment group
operating the CBT prototype system
• Each session was
60 min long
• a
10-min introduction
• a
10-min simulation test
• a
30-min simulation learning
Evaluation and conclusion
Summative evaluation
• Based on the usability goals of
efficiency
and
effectiveness
, we
divided the evaluation analysis into
– system design
usability analysis :
subjective interface
opinions
– learning performance
usability analysis:
Evaluation and conclusion
Summative evaluation
System design usability analysis
• The questionnaire
– designed based on the CAI program software evaluation rules by Du (1991)
– Questions were divided into four constructs of operation, program software, correc tness of material, and feedback handling
– A Likert-type scale ranging from very good (+2), good (+1), average (0), bad (1) t o very bad (2) were used.
• Cronbach coefficient and content validity coefficient proposed by Lin (199 3) were used. Table 4 showed both reliability and validity of the questionnai re were acceptable.
• No significant difference, most values in Table 5 were positive and implied m ost subjects approved the system design
• Input operation, function display, selected examples, and terminology explana tion were rated to be most appropriate
• Item received negative values included volume of images, depth of material, a nd variations of feedback (volume).
Evaluation and conclusion
Summative evaluation
Learning effects evaluation analysis
• The test scores after the learning period showed significant increases. This suggests that the intended simulation tactics in Section 4.1 are useful for assisting learning simulation analysis
• The experiment group did marginally better, the t test applied did not find significant difference
– We suspect that the 30-min learning period may not be long enough to detect possible differences
– The experiment might produce stronger results if the evaluation were conducted with a problem solving format on more realistic cases instead of easily
memorized questions-and-answers format
• The additional questions in the second test indicated that two items
‘‘boring’’ and ‘‘interesting’’ had significant differences between the two groups with a = 0.05.
• Even though the t test showed insignificant difference on the ‘‘convenient’’ item, the data revealed that more subjects in the control group favored the format they used
•
