Vol. 10, 298–308, Fall 2011
Article
Online Teaching Tool Simplifies Faculty Use of Multimedia
and Improves Student Interest and Knowledge in Science
John P. Walsh,
*
†Jerry Chih-Yuan Sun,
*
‡and Michelle Riconscente
§*Davis School of Gerontology and†Program in Neuroscience, University of Southern California, Los Angeles,
CA 90089;‡Institute of Education/Center for Teacher Education, National Chiao Tung University, Hsinchu,
Taiwan; and§Rossier School of Education, University of Southern California, Los Angeles, CA 90089
Submitted March 29, 2011; Revised June 16, 2011; Accepted June 16, 2011 Monitoring Editor: Eric H. Chudler
Digital technologies can improve student interest and knowledge in science. However, researching the vast number of websites devoted to science education and integrating them into undergraduate curricula is time-consuming. We developed an Adobe ColdFusion– and Adobe Flash–based system for simplifying the construction, use, and delivery of electronic educational materials in science. The Online Multimedia Teaching Tool (OMTT) in Neuroscience was constructed from a ColdFusion-based online interface, which reduced the need for programming skills and the time for curriculum development. The OMTT in Neuroscience was used by faculty to enhance their lectures in existing curricula. Students had unlimited online access to encourage user-centered exploration. We found the OMTT was rapidly adapted by multiple professors, and its use by undergraduate students was consistent with the interpretation that the OMTT improved performance on exams and increased interest in the field of neuroscience.
INTRODUCTION
Significant investment has been put forth by funding agen-cies, scientists, and programmers to create what they hope will be digitally engaging curricula for the sciences. A num-ber of educational studies now support the value of using multimedia to enhance lectures to increase student inter-est and knowledge in science (O’Day, 2007). However, mul-timedia by itself does not always lead to critical thinking and imagination in the science learner (Greenfield, 2009). A balance of multimedia, interactive exercises, and traditional texts produces the best outcome for learning in the sciences (Greenfield, 2009). All of the above-indicated educational resources are now available on the Internet; however, the biggest challenge is the time it takes to sift through the vast
DOI: 10.1187/cbe.11-03-0031
Address correspondence to: John P. Walsh (jwalsh@usc.edu). c
2011 J. P. Walsh et al. CBE—Life Sciences Education c 2011 The American Society for Cell Biology. This article is distributed by The American Society for Cell Biology under license from the author(s). It is available to the public under an Attribution– Noncommercial–Share Alike 3.0 Unported Creative Commons License (http://creativecommons.org/licenses/by-nc-sa/3.0). “ASCBR” and “The American Society for Cell BiologyR” are
regis-tered trademarks of The American Society for Cell Biology.
amount of resources available, as well as the time it takes to integrate newfound materials into a cohesive curriculum (Alberts and Mayo, 2009).
Significant resources have been put into developing central repositories for simplifying access to multimedia programs in the hope that faculty will adapt multimedia tools for their courses (Mervis, 2009). The two largest sites are the National Science Digital Library (NSDL; http://nsdl.org) funded by the National Science Foundation (NSF) and the Multime-dia Educational Resource for Learning and Online Teaching (MERLOT; www.merlot.org/merlot/index.htm) supported by both public and private contributions. However, recent studies indicate the NSDL is highly underused, and ques-tions have been raised about the costs and benefits of the pro-gram (Mervis, 2009). Faculty buy-in for use of multimedia is limited by a lack of training and knowledge about the use of the resources and the fear that time spent in such activities will not be recognized by their university during evaluation for promotion (Ruiz et al., 2006). Furthermore, questions exist about the value of many multimedia sites that have not been assessed for their academic impact (Alberts and Mayo, 2009; Mayo, 2009).
We describe an educational tool developed as a platform for simplifying adaptation of the many online resources available worldwide. The Online Multimedia Teaching Tool (OMTT; http://omtt.usc.edu) described in this report accomplished
Figure 1. Screen capture of course-construction interface used to create the “Synapses” chapter. (A) Image of course-construction interface illustrating how banner is uploaded (top arrow) and colors of outline buttons (middle arrow) and border lines (bottom arrow) are selected (note X in color-selection box for each choice). (B) Home page for “Synapses” chapter showing the presized banner uploaded and the color of the main topic tabs and border lines selected by the interface to the left. The introductory video of a USC student telling the user how to use the OMTT and the learning goals of the chapter are also illustrated.
this goal by structuring the program to deliver online re-sources in an outline format as is practiced by most instruc-tors in their lectures (Brown and Manogue, 2001). A survey of users revealed that faculty who used the OMTT in Neuro-science recommended its use. Student engagement in neuro-science and performance on standard exams was positively impacted by use of the OMTT in Neuroscience.
METHODS
OMTT Construction
The OMTT’s format was developed with a dynamic construc-tion, which utilizes a single flexible template that generates the course by placing assets (text, images, videos, interactive exercises, animations, etc.) on the screen once the student accesses it. Asset placement, size, and other appearance in-formation are stored in a database that is accessed by the template depending on a student’s navigational activity. To decrease production costs and to allow for access on all com-puter operating systems, the course uses off-the-shelf devel-opmental software programs found within Adobe’s Creative Suite (currently version 5.5) and ColdFusion. Adobe Photo-shop and Illustrator were utilized for image production, and Adobe Premiere and Adobe AfterEffects were used for post-ing and editpost-ing video clips. Adobe Flash was used to develop highly interactive assets and the course’s template. Dynamic course design was developed by using the database manage-ment software ColdFusion to transfer all necessary informa-tion from a Structured Query Language (SQL) database to the course’s Flash template. Using commonly available soft-ware like the products listed, the OMTT course and course-development sites can be accessed via the Internet with any computer that has the free Adobe Flash plug-in installed.
The course-construction interface, programmed in Cold-Fusion, was developed to reduce the cost and time for course
development. The construction site allows for the building of each course’s template (i.e., the outline format of main topics, subtopics, and buttons listed at the bottom of the course and the chapter banner) and pages. The intuitive course-construction interface cues the developer to place ti-tles into each main and subtopic site, and then offers a menu of page designs. Within each page design, the site prompts for the uploading of assets (images, videos, animations, and links) directly into each page (Figures 1 and 2). The course-construction interface not only simplified the course-construction of each chapter but, of equal importance, it also eased modifi-cation of OMTT content at any time during the process. Stu-dent and professor registration sites for tracking of learning, also programmed in ColdFusion, allowed for instantaneous course access through the automatic generation of individual user names and passwords. This feature of the OMTT allowed for anonymous oversight of student performance on midterm and final exams, in which students were coded (1-user or 2-nonuser) with a binary designation. ColdFusion program-ming also allowed us to track student use by counting the number of clicks (times) students accessed the main topics listed at the top of each instruction chapter during the 2009– 2010 academic year. The technical team assembled to pro-duce the OMTT consisted of an Adobe Flash programmer, an Adobe ColdFusion programmer, a content manager, and an assessment manager. The OMTT construction required 25% effort (3 calendar mo/yr) for the Adobe Flash and Adobe ColdFusion programmers over a 3-yr period. The content manager’s and the evaluator’s work equaled 2 and 1 mo ef-fort, respectively, over the same construction period.
The OMTT in Neuroscience was released in three phases. The beta module on Parkinson’s disease was the first chap-ter, made available in 2008, and was well received by stu-dents and instructors who used it in undergraduate neuro-science courses (Misiaszek et al., 2008). The following year, we released OMTT chapters entitled “Ion Channels” and “Synapses.” Chapters entitled “Functional Neuroanatomy”
Figure 2. Screen capture of course-construction interface used to create “Functional Neuroanatomy” chapter. (A) Image of course-construction interface illustrating how presized images and their link to educational exercises are loaded into each template or page (see bottom browse buttons). This part of the interface, selected for by a prior key selection in the interface, allows a faculty user to load assets or exercises into a single subtopic page. The template design is chosen from prefabricated designs or is custom built (red boxes in center of interface). (B) Screen capture of the subtopic page entitled Auditory and Language Processing found under the main heading Cortical Specialization (blue main tab button) in the “Functional Neuroanatomy” chapter built with the interface shown in (A). Three separate educational exercises, with descriptive bullets above and developer credit below the exercise, are shown.
and “Biogenic Amines” were released during the most recent year and, at the same time, the “Parkinson’s Disease” chap-ter was reformatted to more closely match the new chapchap-ters developed during the 2-yr, NSF-funded project. Each OMTT chapter was designed to have a logical flow for both instructor in-class use and for students to navigate on their own. Chap-ter design for core subjects in neuroscience, like ion channels, synapses, and functional neuroanatomy followed the same format, starting with basic information in the far left main tab and then sequentially offering more complex information as the user moved to the right. Clicking on a main tab re-vealed a number of subheadings or subtabs to choose from under the main tab topic. The organization of the subtab top-ics was to start at the topic with more basic information; the user selected increasingly more challenging areas by moving down through the subtabs. For example, in the “Synapses” chapter, the left-to-right sequence of main tabs to select from are: 1) Presynaptic Terminal, 2) Postsynaptic Mechanisms, 3) Complex Circuits Between Neurons, and 4) Synaptic Plastic-ity. Within the first main tab, Presynaptic Terminal, the user can select from the following top-to-bottom subtabs: 1) Ba-sics of Neurotransmitter Release, 2) Molecular Mechanisms of Neurotransmitter Release, 3) Article on Neurotransmit-ter Release, 4) NeurotransmitNeurotransmit-ter Textbook, and 5) Scientists Studying Neurotransmitter Release. Specialty chapters like “Parkinson’s Disease” and “Biogenic Amines” follow a dif-ferent format that is better suited to the unique features of these fields of study. Each OMTT chapter has two final main tabs designed to encourage students to go outside the class-room to expand their interest in neuroscience: main tab 5, entitled Laboratory and Readings, and main tab 6, entitled Research Opportunities.
The OMTT could only be accessed via student and in-structor log-in during the first 2 yr of development, which allowed us to track user navigation patterns and user as-sessment of the OMTT via an imbedded Qualtrics survey
(www.qualtrics.com). A front-end home page for the OMTT was created after this initial rollout using a content manage-ment system. The new OMTT home page provides many resources on digital educational practices, links to sites of-fering career-oriented interviews from neuroscience faculty, and collections of other multimedia resources. It also serves as a launching point that allows the user to choose between open access or log-in options (Figure 3). The log-in option was retained for those faculty interested in tracking how their students navigate the course, how their students perform on built-in quizzes, and what their students’ outcomes are; all of this information is available from the embedded Qualtrics course-assessment survey.
Participants and Study Design
Student Participants. The OMTT in Neuroscience was used
to supplement existing curricula in a number of courses. Each course used established textbooks and sets of support-ive PowerPoint slides developed during previous semesters. Professors used the OMTT in class, when appropriate to sup-plement their curricula, and registered students had online access at all times. We used a quasi-experimental design, where student use of the OMTT and participation in the study was completely voluntary (Shadish, 2006; Crippen and Earl, 2007). This research method involved comparing outcomes for students who volunteered to participate in the program (experimental group) relative to a “matched” control group of students in their class who did not elect to register for use of the course. The major ethical advantage of this research design is that it allows all students who express interest in experiencing the program to have access to it, thus circum-venting the problem associated with an experimental design in which some students are arbitrarily denied program access to serve as a control group (Cuseo, 2007).
Figure 3. Screen capture of the OMTT in Neuroscience home page. Image illustrates the home page of the OMTT construction with a content management system that connects to the five Adobe Flash chapters found in the OMTT (“Ion Channels,” “Synapses,” “Func-tional Neuroanatomy,” “Parkinson’s Dis-ease,” and “Biogenic Amines”). Clicking on the Enter Course button takes you to a page where you choose log-in or open access routes to enter the course. The home page of the OMTT has many links to neuroscience and multimedia resources, an instruction manual, and an embedded link to Qualtrics surveys.
However, one methodological disadvantage of the quasi-experimental design is that students are not randomly as-signed to experimental and control groups, as they are in a true experimental design (hence its name, “quasi-experimental”). Consequently, this design fails to control for volunteer bias, leaving open the possibility that positive out-comes resulting from program participation may be due to the highly motivated nature of students who elect to partici-pate in the program, rather than to the effects of the program itself. Another limitation of the study design is that data were not collected regarding participant demographics or other variables that might affect learning. Thus, it was not possible to determine the equivalency of the experimental and control populations. To address this limitation, we compared par-ticipant and nonparpar-ticipant results related and unrelated to OMTT content. It has been shown that the combination of us-ing a quasi-experimental design and not havus-ing demographic data on student users constitutes a contextual factor that limits the boundaries for generalizing outcomes (Shavelson and Towne, 2002). We also took care not to make causal claims between OMTT use and exam performance due solely to as-sociations (Shavelson and Towne, 2002).
The education study was divided into two components. First, we conducted a survey of user opinion on use of the OMTT, which included a large sample of students from mul-tiple academic institutions across the United States and a small sample of professors who either used the OMTT in class or tried it out on their own. The questions in each survey were developed to assess engagement in using the
OMTT and in the field of neuroscience. The second compo-nent of the study examined whether OMTT use was associ-ated with better performance on traditional in-class forms of assessment of student knowledge in neuroscience (i.e., midterm and final exams). As indicated previously, student use of the OMTT was completely voluntary, and group as-signment was based on whether the student had registered for the OMTT in Neuroscience or not. Registered students and faculty gained access to the OMTT via a personalized user name and password created during registration. This design meant performance in some courses could not be com-pared, due to the large percentage of students who registered. Two courses, however, produced a good balance of OMTT-registered users and nonusers, which enabled our compar-ison of performance on OMTT-relevant midterm and final exams. One course was a large biology class at the Univer-sity of Southern California (USC; BISC 421) and the other was a small psychology class at California State University at San Bernardino (CSUSB; Introduction to Psychopharma-cology, Psychology 433). The largest sample came from BISC 421, with 63 registered student users and 99 students elect-ing not to register for use. The OMTT was not used in BISC 421 for material covered in the first midterm, but it was used for material covered in the second midterm and the final. BISC 421 students were also evaluated for their performance during their in-class oral presentation. Student performance on oral presentations was assessed by the students’ demon-stration of mastery of specific neuroscientific concepts, the ability to connect these principles to other concepts, and
finally, their ability to explain these concepts and connec-tions in a clear and coherent way. Student performance on exams was evaluated similarly, with questions designed to test student ability to solve problems rather than simply re-peat basic concepts. For instance, instead of asking “Which ion species is responsible for the depolarization?,” we asked in one exam, “What will happen if we block Na+transport into the cell?” or “How do inhibitory and excitatory synapses interact to mold the output of a nerve cell?” Teaching assis-tants graded the midterms and final exam in BISC 421. They were blind to the students’ participation in the OMTT. Proce-dures were approved by the USC Institutional Review Board on Human Subjects. Informed consent was received from all participants, and the informed consent form guaranteed con-fidentiality and anonymity.
Faculty Participants. The original OMTT was built for
test-ing at USC. However, the online nature of the OMTT meant we had the opportunity for immediate and widespread dis-semination of chapters as they were produced. Professors and students from biology, engineering, and gerontology courses at USC; psychology courses at CSUSB, Gettysburg College in Pennsylvania, and Central Michigan University; and a neu-roscience course at Tulane University in Louisiana were able to use the “Ion Channels” and “Synapses” chapters of the OMTT. The modular nature of the product we developed meant we were able to advertise components of the total project outlined for production as they became available.
We also announced the rollout of the OMTT in neurobiol-ogy to a large audience of Society for Neuroscience educators, the Faculty for Undergraduate Neuroscience (FUN) educa-tion, via the FUN listserv before the society’s annual inter-national meeting, which was held in Chicago in November of 2009. This resulted in many new registrations in the week following the meeting. Ten faculty who registered with the OMTT after the Society for Neuroscience meeting were asked to sample the OMTT and fill out our Qualtrics survey. Five faculty agreed, and the outcomes of their responses, along with those from faculty who used the OMTT for instruction, are reported in our analysis (Table 1).
RESULTS
Course Performance Data
The OMTT was used in a large neuroscience course taught by the Department of Biological Sciences at USC (BISC 421– Neurobiology) during the Fall semester of 2009; this course had an enrollment of 162 students. The OMTT website was presented to students in their course materials, and students were asked to register for its use on a voluntary basis. Sixty-three students registered to use the OMTT and 99 students did not. Instructors used the OMTT to support their existing lectures for sections of the course covering ion channels, cel-lular potentials, and synapses. Student assessment in BISC 421 was based on two midterms, a presentation in the course discussion section, and a final exam. The OMTT was not used for material covered in the first midterm, but it was used for material covered in the second midterm and the final exam. Student performance was not different between registered and nonregistered students for the first midterm (registered, 85.82 ± 1.31; nonregistered, 83.6 ± 1.46; two-tailed t test,
p= 0.14). Student performance was also not different between registered and nonregistered students for their presentations (registered, 48.54; nonregistered, 48.25; two-tailed t test, p= 0.15; Figure 4).
Registered students did significantly better on exams, how-ever, when the instructor used the OMTT in lectures, and the exams covered material present in the OMTT (Figure 1). Test scores for the second midterm were 78.94± 1.42 for registered students and 73.17± 1.29 (± SEM) for nonregistered students (two-tailed t test, p< 0.004). Scores on the final exam were 116.52± 2.76 for registered students and 107.06 ± 2.27 (± SEM) for nonregistered students (two-tailed t test, p< 0.009). Registered students ultimately accumulated more total points and received a better grade in the class (registered, 351.6± 5.4 [± SEM]; nonregistered, 331.04 ± 5.7 [± SEM]; two-tailed
t test, p< 0.0075; Figure 4).
The OMTT was also made available on a voluntary basis to students enrolled in Psychology 433 at CSUSB, PSYCH 236 at Gettysburg College, and GERO 414 at USC. Over 90% of USC (GERO 414) and Gettysburg College students registered, so no comparison could be made between registered and non-registered students for exam grades when the OMTT was used. CSUSB’s class size was 16 students, with five registering and 11 not registering. Three exams were given to the CSUSB students, and the OMTT in Neuroscience was used to sup-plement lectures in two of the exams. While not significant, a trend was seen for better performance by OMTT-registered students for the first two exams, where the instructor used the OMTT. The third exam was on material not covered in the OMTT, and no difference in performance was seen between the two groups. Exam score averages for the first midterm, which included OMTT use, were 140.4± 5.36 (± SEM; n = 5) for registered students and 118.23± 11.4 (± SEM; n = 11) for nonregistered students (two-tailed t test, p= 0.10). Exam scores for the second midterm, which also included OMTT use, were 152.6± 7.5 (± SEM; n = 5) for registered students and 136.7± 8.0 (± SEM; n = 11) for nonregistered students (two-tailed t test, p= 0.10). Exam scores for the third midterm, which did not include OMTT use, were 273.8± 26.5 (± SEM;
n= 5) for registered students and 253.8 ± 26.2 (± SEM; n = 11) for nonregistered students (two-tailed t test, p= 0.54).
OMTT Use Data
ColdFusion programming built into the OMTT allowed us to monitor the use or navigation patterns of logged-in stu-dents during the testing period (Figure 2). The majority of students using the OMTT for Fall 2009 came from the USC course (BISC 421). Click frequency confirmed that most un-dergraduate students focused on the basic information found in the left main tabs (see bar graphs for “Ion Channels,” “Synapses,” and “Parkinson’s Disease” chapters in Figure 5). Chapter design is learner-centered, with more complex in-formation being provided in each successive main tab to the right. While the frequency of students investigating the labo-ratory, primary reading, and research opportunity main tabs was lower, the distribution revealed interested students did access these areas. The neuroscience program at USC has a large focus group of faculty interested in synaptic plasticity, and this area is stressed in most neuroscience courses. The click frequency in the “Synapse” chapter confirmed this fo-cus, with a large number of searches being performed in the
Table 1. OMTT in Neuroscience faculty survey questionsa Na¨ıve faculty (n= 5) Faculty users from Table 2 (n= 5) Using the OMTT on their own time will foster my students’ interest in science 3.0 3.6 Students will find using the OMTT on their own time to be more trouble than it is worth 1.8 1.6 Using the OMTT on their own time will make it easier for my students to grasp scientific concepts 3.4 3.6 My use of the OMTT should help students see how what we cover in class connects to real scientific research 2.8 3.2 If I were to use the OMTT, students would want to use the OMTT outside of class 3.0 4 Based on what I’ve seen so far of the OMTT, I would use the OMTT during class lecture 3.4 4 Based on what I’ve seen so far, I would encourage my students to use the OMTT on their own time 3.6 4 Based on what I’ve seen so far, I would recommend the OMTT to colleagues 3.6 4 Based on what I’ve seen so far, I would like to see more OMTT modules developed 3.4 4
I found the OMTT easy to use in my lectures — 3.8
I found the OMTT complemented my existing curriculum in neuroscience — 4
aFour-point Likert scale: very unlikely (4), unlikely (3), likely (2), very likely (1), where 2.5 would represent the middle.
synaptic plasticity main tab (main tab 4; Figure 5, red oval). The “Parkinson’s Disease” chapter is a specialty area and was stressed in the smaller GERO 414 course at USC, which likely explains its lower frequency of use. Interestingly, use patterns in the “Parkinson’s Disease” focused on the biology-oriented main tab Etiology, which was stressed in class (Figure 5, red oval).
OMTT Survey Data
Students from two USC courses (BISC 421, GERO 414), one Gettysburg College course (PSYCH 236), and one CSUSB course (PSYCH 433) were asked to take a survey about the OMTT in Neuroscience at the end of the Fall 2009 semester.
Student opinion about the OMTT in Neuroscience was most positive in USC courses , where students uniformly found the OMTT increased their interest in neuroscience and their ap-preciation of how research creates knowledge in neuroscience (Table 2). Survey scores were compared with a middle value of 3.5 (two-tailed t test assuming unequal variances). Similar trends were seen at Gettysburg College, but CSUSB students were less positive. The sample size from CSUSB was quite low (n= 8), which may have impacted the outcome. In general, students also found faculty use of the OMTT in Neuroscience in lectures to be a positive, and that it increased their overall interest. Again, the greatest effect was seen at USC, where in-structors worked with OMTT developers. The professors at Gettysburg and CSUSB were simply provided the URL and
Figure 4. Grade distribution of registered users of OMTT and control group who did not use the OMTT for a large undergraduate neuroscience course at USC. Red bars repre-sent registered users of OMTT and blue bars represent students who elected not to use the OMTT in BISC 421–Neuroscience course at USC. Gray highlight box surrounds assess-ments not associated with OMTT use. *p< 0.005, t test assuming unequal variance.
Figure 5. Patterns of OMTT navigation produced during sampling period collected via Adobe ColdFusion database. Frequency of clicks on each main topic are indicated and their associated subheading cat-egories are shown for the “Ion Channels,” “Synapses,” and “Parkin-son’s Disease” chapters. Red ovals highlight unique use patterns seen in the “Synapses” and “Parkinson’s Disease” chapters.
asked to consider using the program. Two negative control questions are listed at the bottom of Table 2. One question controlled for the student view of faculty use of the OMTT in class, and the outcome supports the earlier finding that
students found faculty use of the OMTT in class to be a pos-itive. The second question addressed the issue of whether students felt using the OMTT was a practical use of their time, which is addressed in the Discussion. Faculty who used the OMTT in Neuroscience (indicated in Table 2) were surveyed about their experience. The entire faculty felt the OMTT was easy to use and that the information added value to their existing curricula (Table 1). A small sample of additional faculty from universities other than USC, who had never seen the OMTT in Neuroscience, were asked to look at the OMTT during the Fall 2010 semester and to take a survey (na¨ıve faculty in Table 1). The five na¨ıve faculty, on average, expressed a very positive review of the OMTT in Neuroscience (Table 1). Both sets of faculty felt it was/would be useful for their students and that the OMTT would help students make a better connection between sci-ence facts and the research used to get the facts. On average, the faculty who actually used the OMTT in the 2009/2010 academic year had a better view of the OMTT than the na¨ıve faculty sample. The questionnaire used a 4-point Likert scale of 1= very unlikely, 2 = unlikely, 3 = likely, and 4 = very likely, where 2.5 would be considered the middle.
Each OMTT survey ended with an opportunity for par-ticipants to send open-ended comments that were used immediately as formative assessment. The original rollout of the OMTT began with a bland log-in page used to gain access to individual chapters. Faculty and students alike strongly emphasized the need to create an engag-ing and informative home page, a task that was accom-plished using a content management system for connect-ing to the original Adobe Flash instructional chapters. Fac-ulty and students also suggested we make the course chap-ters completely open access and leave the log-in feature available for those who might be interested in this option. Faculty and students asked for the development of more instructional chapters beyond the initial offering of “Ion Channels” and “Synapses.” Subsequently, three new stand-alone chapters were added (“Functional Neuroanatomy,” “Biogenic Amines,” and a revised “Parkinson’s Disease”). Students suggested interactive quizzes be added to each chapter; this is now a feature in each chapter. Instructional bullets were added to each asset or learning exercise in re-sponse to student requests. The original rollout of the OMTT in Neuroscience used an outside vendor for its Adobe Flash video server that was cost-effective but slow. Complaints from faculty and students alike on download speed pointed us to the use of YouTube as an inexpensive and reliable place to house OMTT videos. Students suggested more students would use the OMTT if professors would make its use part of the grade they received in the class. Students and faculty both indicated greater use of the OMTT would occur if the curricula of the OMTT matched the curricula of already es-tablished courses, and if it was demonstrated that the OMTT improved student learning and interest in neuroscience.
DISCUSSION
The OMTT in Neuroscience was developed to increase stu-dent knowledge and interest in neuroscience and to simplify instructor adaptation by providing a unique lecture-based method of digital curricula delivery. Key aspects of the design were 24/7 accessibility via the Internet and ease of adaptation
Table 2. OMTT in Neuroscience student survey questionsa USC BISC 421 (33) USC GERO 414 (25) Gettysburg PSYCH 236 (18) CSUSB PSYC 433 (8) Sum of all classes (84) The OMTT made me more interested in
neuroscience
3.9± 0.2,
p< 0.06 4.5p< 0.001± 0.2, p3.6< 0.55± 0.3, 3.1p< 0.55± 0.5, 4.0p< 0.001± 0.1, The OMTT made it easier for me to
understand scientific concepts
4.3± 0.2, p< 0.001 4.9± 0.2, p< 0.001 4.2± 0.2, p< 0.001 3.7± 0.4, p< 0.61 4.4± 0.1, p< 0.001 The OMTT helped me do better in this
course 3.7± 0.2, p< 0.31 4.1± 0.2, p< 0.001 3.6± 0.2, p< 0.7 3.5± 0.5, p= 1.0 3.8± 0.1, p< 0.021 The OMTT helped me see how what we
learned in class relates to what scientists do for work
4.2± 0.2, p< 0.001 4.5± 0.2, p< 0.001 4.0± 0.2, p< 0.051 3.7± 0.4, p< 0.61 4.2± 0.1, p< 0.001 I wish more courses included the OMTT 4.1± 0.2,
p< 0.001 4.6p< 0.001± 0.2, p3.7< 0.41± 0.2, 3.5p= 1.0± 0.5, 4.1p< 0.001± 0.1, The instructor’s use of the OMTT in class
helped me understand the material better 4.3± 0.2, p< 0.001 5.0± 0.2, p< 0.001 4.1± 0.2, p< 0.011 3.7± 0.5, p< 0.71 4.4± 0.1, p< 0.001 The instructor’s use of the OMTT in class
helped me know how what we learned in class connected to scientific research
3.8± 0.2,
p< 0.091 4.2p< 0.001± 0.2, p3.4< 0.71± 0.2, 3.3p< 0.71± 0.4, 3.8p< 0.012± 0.1, The instructor’s use of the OMTT made
me want to use the OMTT outside of class 4.0± 0.2, p< 0.011 4.4± 0.2, p< 0.001 3.6± 0.3, p< 0.71 3.5± 0.5, p= 1.0 4.0± 0.1, p< 0.001 Using the OMTT was not a good use of
my time
4.4± 0.3,
p< 0.011 2.5p< 0.11± 0.5, p4.5< 0.001± 0.2, 5.1p< 0.041± 1.0, 4.0p< 0.001± 0.2, The instructor’s use of the OMTT made
me more confused 2.9± 0.2, p< 0.001 2.3± 0.9, p< 0.001 2.5± 0.2, p< 0.001 2.3± 0.8, p< 0.001 2.4± 0.1, p< 0.001
aSix-point Likert scale: strongly agree (6), agree (5), somewhat agree (4), somewhat disagree (3), disagree (2), strongly disagree (1), where 3.5
would represent the middle. Values in parentheses indicate sample size for each class. p values indicated below mean± SEM for each question are from t tests assuming unequal variance when values were compared with the 3.5 middle value of the 6-point Likert scale.
by instructors to enhance their existing curricula. Other im-portant goals sought in the development of the OMTT were to create a means for streamlining use of the many digital resources already available online, and to do so in a cost-effective way. This latter goal was from two USC achieved through the design of an online course-construction inter-face that easily incorporated and displayed these resources in a logical course outline format used by most instructors during lectures (Brown and Manogue, 2001; Figure 6).
The OMTT was assessed for its impact on student per-formance on in-class exams, student attitudes about neuro-science and research, and faculty views on its ease of use in the classroom and the value it added to their courses. The initial offering of the OMTT in Neuroscience required students and faculty to register for its use, and students were told registra-tion was on a voluntary basis and not required for their course evaluation (quasi-experimental design). Four courses used the OMTT during the 2009–2010 academic year. Two courses had over 90% OMTT registration, which meant grade com-parison between registered users of the OMTT and nonusers could not be performed. Two other courses, a large biology class at USC (BISC 421) and a small psychology class at CSUSB (PSYC 433), produced distributions that allowed compari-son between registered versus nonregistered students. Out-comes from these two courses were consistent with the in-terpretation that using the OMTT in Neuroscience was asso-ciated with better performance on traditional in-class exams (Figure 4). While this outcome is promising, the interpreta-tion of the data is guarded. The quasi-experimental design of the study means we cannot unambiguously attribute the
differences in exam performance to OMTT use, as registered students may have been a student population who would have done better in the course irrespective of their use of the OMTT in Neuroscience (Crippen and Earl, 2007). An-other important limitation with the quasi-experimental de-sign we used is the fact that the association between op-tional use of the OMTT and course performance does not infer cause (Shavelson and Towne, 2002). However, if stu-dent motivation or other stustu-dent variables were indeed re-sponsible for the improved exam scores of participating stu-dents, one might expect participating students to perform better on the exams unrelated to OMTT content, and they did not. Nonetheless, use of a quasi-experimental design is an effective means for the initial testing of educational inno-vations, and this approach circumvents the ethical problem associated with arbitrarily denying students access to a tool that could improve their grades (Shadish, 2006; Cuseo, 2007; Walker et al., 2008; Carle et al., 2009). The quasi-experimental design limitations previously discussed also apply to the outcomes from the survey of student attitudes about the OMTT.
The OMTT in Neuroscience was viewed positively by stu-dents, on average across all courses, as measured by multiple levels of engagement (Table 2). This outcome included stu-dents being more interested in neuroscience, stustu-dents having a better understanding of scientific concepts, and students having a better understanding of how research connects to the curricula being taught. On average, students also felt the instructors’ use of the OMTT in Neuroscience in class helped them understand the material better and made them want
Figure 6. Screen capture of “Parkinson’s Dis-ease” chapter illustrating the outline format used to design the OMTT in Neuroscience. The Symptoms/Treatments main tab has been selected, which produces a drop-down menu of subcategories to choose from under the se-lected main tab. The Symptoms of PD sub-tab was chosen for this screen capture, which shows a video of a patient suffering from Parkinson’s disease.
to investigate the OMTT independently outside class. This measure was validated by the negative construct question “The instructors’ use of the OMTT made me more confused,” which was significant in the negative direction (Table 2). Students also expressed an interest in having more classes adapt the use of the OMTT. Analysis of the breakdown of results based upon institution (i.e., USC, Gettysburg College, and CSUSB) revealed students at USC were more positive than students from the other two outside institutions. These differences might be explained by the small sample size for non-USC courses, which reduces the power of the analysis, and by the lack of faculty development in the OMTT’s initial rollout. Instructors were simply provided the URL, which re-quired them to log in. These instructors were asked to adopt OMTT to supplement their existing curricula, and no fur-ther instructions or faculty development were supplied. By contrast, a professor at Central Michigan University was pro-vided a specific URL for the “Parkinson’s Disease” chapter from the OMTT in 2008, with explicit instructions on its use, and student perception was extremely positive (Misiaszek
et al., 2008). The front end of the OMTT, developed through a
content management system, was constructed after the 2009– 2010 academic year and, in response to formative assessment, this area now contains a clear set of instructions on how to use the OMTT, as well as links to other useful education resources. Studies have shown dissemination and use are enhanced if a faculty development component is included when introduc-ing innovation in teachintroduc-ing, and it is likely that this component would have improved OMTT use and, consequently, survey outcomes (Farmer, 2004; Steinert et al., 2006). Improved fac-ulty use and integration of the OMTT into existing curricula would make students feel their use of OMTT impacted their class grade, leading to positive student response to the value of OMTT and the time expended on its use (Table 2; Jones and Brickner, 1996). Indeed, open-ended comments about the
OMTT by students stated they would have been more moti-vated to use the OMTT if they knew it counted toward their grade.
Two different groups of faculty were assessed for their at-titudes about the use of the OMTT. The first group was those faculty who used the OMTT in the 2009–2010 academic year (Table 2), and the second group was a random group of fac-ulty registrants from across the country (na¨ıve facfac-ulty). Both sets of faculty were positive about the use of the OMTT to supplement their lectures, with those faculty who actually had the chance to use the OMTT during the 2009–2010 aca-demic year having slightly more positive views. These data confirm the OMTT can be easily adapted for use in lecture by faculty, and that a faculty development component will ease its use and bring greater appreciation of its value.
Our analysis of use patterns of the OMTT, as determined by the number of navigational clicks under each main topic or tab found in each chapter, suggested students tended to focus on the more basic instruction found in the beginning or left-hand main tabs. A few differences were observed, how-ever. The “Synapses” chapter showed a larger distribution of navigational clicks in the Synaptic Plasticity main topic. The likely explanation for this trend is that instructors stressed this area, as synaptic plasticity is an area of expertise and in-terest for many USC instructors, and the USC course biased the sample with its larger course enrollment (Figure 5). The “Parkinson’s Disease” chapter showed a larger distribution of navigational clicks in the Etiology main topic or tab, likely reflecting the biological focus of this subject heading.
Open-ended comments from students and faculty collected via Qualtrics surveys were used as formative assessment to modify the OMTT. The first iteration of the OMTT began with a stock log-in page that launched three instructional chapters as of Spring 2010. The log-in page also con-tained hypertext linking to our Qualtrics survey. Open-ended
comments from the surveys led to the development of an attractive and engaging home page for launching OMTT, the creation of a completely open access non-log-in path-way to instructional chapters, the use of bullets to describe each instructional component or asset, the adaptation of YouTube for more rapid video streaming (Figures 1 and 6), and the development of chapter quizzes for each chapter. Access to the OMTT was modeled after the approach taken by the Open Learning Initiative at Carnegie Mellon Univer-sity (http://oli.web.cmu.edu/openlearning/index.php) and other universities where students and faculty can use courses with or without registration (Centivany and Glushko, 2010). An important feature of the OMTT, however, is that faculty
and students are the target audience. The OMTT does not
have to exist as a stand-alone course. It can be used by any instructor worldwide to augment an existing curriculum. The OMTT was constructed to provide a complete multimedia ex-perience for studying neuroscience. Many studies have doc-umented the value of multimedia in science education for engaging the learner and creating a deeper understanding and knowledge about the subject (O’Day, 2007). The results from this study support this view. However, not all learners’ needs are necessarily met by multimedia alone, with tradi-tional text being an essential component for some learners to develop imagination, critical thinking, and deeper learn-ing (Greenfield, 2009). The OMTT was structured to provide complementary digital resources for each topic of interest, including videos, animations, interactive exercises, and what are referred to in the OMTT as online textbooks. We were able to achieve this goal for an entire semester’s worth of neuroscience curriculum by soliciting collaboration with de-velopers of digital resources via email and, with their agree-ment, citation of each asset placed in the OMTT in Neuro-science. These steps were taken to ensure the OMTT com-plied with copyright issues (Centivany and Glushko, 2010). Online readings from primary literature are also provided, and comply with copyright issues by using links to locations for these articles within publishers’ websites. Thus, access to the research articles requires using a computer that verifies journal subscription (i.e., a university computer). This feature enables faculty to use literature to teach, and it allows moti-vated students to use literature to learn, which is important, as studies indicate that undergraduates 1) are intimidated by readings from scientific journals, and 2) need to be trained to learn how to use this form of scholarship for career and per-sonal development (Tenopir et al., 2003; Bogucka and Wood, 2009). Analysis of use patterns by students during the 2009– 2010 academic year indicated a small population of moti-vated students did browse the readings offered via these links (Figure 5).
Each chapter in OMTT has its own home page, which au-tomatically loads a student video that provides instructions on the use of the OMTT and the learning goals of the chapter. Online educational tools work best if objectives and learning goals are clearly laid out for the student from the very begin-ning (Swan, 2003). We purposely chose students from groups underrepresented in science for each introductory video. Cole and Espinoza (2008) have found ethnic minorities who iden-tify with members of their own community in science do better, and our design was intended to provide peer identifi-cation as a first step for drawing students from these groups into the OMTT. Increasing access and interest of students
from underrepresented groups is a guideline set forth by the NSF as part of a national initiative for recruiting more stu-dents from underrepresented groups into science (Garelik, 2000; Bianchini et al. 2002).
In closing, we describe a platform that efficiently deliv-ers online digital educational exercises in science. The OMTT in Neuroscience is a demonstration project for an approach that could be applied to any field of science. The course-construction interface used to create the OMTT in Neuro-science proved to be a cost-effective and efficient way of delivering the best practices of cyber-education tools in a structured and complementary manner (Alberts and Mayo, 2009). Examples of best practices in neuroscience educa-tion included The Brain from Top to Bottom from McGill University (http://thebrain.mcgill.ca/flash/index_a.html), animations offered by Sumanus Corporation (www .sumanasinc.com/webcontent/animation.html), and the ex-cellent interactive exercises developed by Tutis Vilis (www.tutis.ca), which were all described in an earlier report by Liu (2006). Mervis (2009) described failures in the use of tools stored in the NSDL. The OMTT is one solution for maxi-mizing the use of the many digital resources available world-wide. Furthermore, the modular structure of the OMTT and its assembly via the course-construction interface simplifies the removal and addition of learning exercises, which pro-vides a cost-effective solution for keeping the content found in the OMTT current (Hines et al., 2009). This latter feature of the OMTT becomes increasingly important in the context of what Mayo (2009) referred to as the inherently unsustain-able model for financing digital curricula development in the academic world.
ACKNOWLEDGMENTS
This study was supported by a grant from the NSF Course Curricu-lum and Laboratory Improvement Program (Award No. 0836966).
REFERENCES
Alberts B, Mayo M (2009). Video: education and technology. Science 323, 53.
Bianchini JA, Whitney DJ, Breton TD, Hilton-Brown BA (2002). To-ward inclusive science education: university scientists’ views of stu-dents, instructional practices, and the nature of science. Sci Educ 86, 42–78.
Bogucka R, Wood E (2009). How to read scientific research ar-ticles: a hands-on classroom exercise. Issues Sci Technol Libr. www.istl.org/09-fall/article4.html (accessed 3 July 2011).
Brown G, Manogue M (2001). AMEE Medical Education Guide No. 22: Refreshing lecturing: a guide for lectures. Med Teacher 23, 231– 244.
Carle AC, Jaffee D, Miller D (2009). Engaging college science stu-dents and changing academic achievement with technology: a quasi-experimental preliminary investigation. Comput Educ 52, 376– 380.
Centivany A, Glushko B (2010). Open Educational Resources and the University: Law, Technology, and Magical Thinking (September 21, 2010). http://ssrn.com/abstract=1680562 (accessed 3 July 2011). Cole D, Espinoza A (2008). Examining the academic success of Latino students in science technology engineering and mathematics (STEM) majors. J Coll Stud Dev 49, 285–300.
Crippen KJ, Earl BL (2007). The impact of web-based worked ex-amples and self-explanation on performance, problem solving, and self-efficacy. Comput Educ 49, 809–821.
Cuseo J (2007). The empirical case against large class size: adverse effects on the teaching, learning, and retention of first-year students. J Fac Dev 21, 5–21.
Farmer EA (2004). Faculty development for problem-based learning. Eur J Dent Educ 8, 59–66.
Garelik G (2000). Report urges scientists to recruit, retain more women and minorities. BioSci 50, 962–962.
Greenfield PM (2009). Technology and informal education: what is taught, what is learned. Science 323, 69–71.
Hines PJ, Jasney BR, Mervis J (2009). Adding a T to the three R’s. Science 323, 53.
Jones JD, Brickner D (1996). Implementation of cooperative learning in a large enrollment basic-mechanics course. Paper presented at the American Society for Engineering Education Annual Conference, 26– 29 June 1996, Washington, DC.
Liu D (2006). WWW: Neuroscience web sites. CBE Life Sci Educ 5, 94–98.
Mayo MJ (2009). Video games: a route to large-scale STEM education. Science 323, 79–82.
Mervis J (2009). NSF rethinks its digital library. Science 323, 54–58. Misiaszek G, Henke M, Riconscente M, Walsh JP (2008). Online mul-timedia teaching tool for Parkinson’s disease. J Undergrad Neurosci Educ (JUNE) 6, A68–A73.
O’Day DH (2007). The value of animations in biology teaching: a study of long-term memory retention. CBE Life Sci Educ 6, 217– 223.
Ruiz JG, Mintzer MJ, Leipzig RM (2006). The impact of e-learning in medical education. Acad Med 81, 207–212.
Shadish WR (2006). Critical thinking in quasi-experimentation. In: Critical Thinking in Psychology, ed. RJ Sternberg, H Roediger, and D Halpern, Washington, DC: American Psychological Association, 37–53.
Shavelson RJ, Towne L (2002). Scientific Research in Education, Washington, DC: National Research Council, National Academies Press.
Steinert Y, Mann K, Centeno A, Doman D, Spencer J, Gelula M, Prideaux D (2006). A systematic review of faculty develop-ment initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach 28, 497– 526.
Swan K (2003). Learning effectiveness: what the research tells us. In: Elements of Quality Online Education, Practice and Direction, ed. J Bourne and JC Moore, Needham, MA: Sloan Center for Online Education, 13–45.
Tenopir C, Pollard R, Wang P, Greene D, Kline E, Krummen J, Kirk R (2003). Undergraduate science students and elec-tronic scholarly journals. Proc Am Soc Inf Sci Technol 40, 291– 297.
Walker JD, Cotner SH, Baepler PM, Decker MD (2008). A delicate balance: integrating active learning into a large lecture course. CBE Life Sci Educ 7, 361–367.
