Effects of student characteristics and question design on Internet search results usage in a Taiwanese classroom

(1)

Effects of student characteristics and question design on

Internet search results usage in a Taiwanese classroom

Chuen-Tsai Sun, Shu-Hao Ye

*

_{, Hsi-Chien Hsieh}

Department of Computer Science, National Chiao Tung University, 1001 University Road, Hsinchu 30010, Taiwan, ROC

a r t i c l e i n f o

Article history:

Received 8 March 2014 Received in revised form 27 April 2014

Accepted 28 April 2014 Available online 9 May 2014 Keywords:

Human–computer interface Interactive learning environments Media in education

Multimedia/hypermedia systems Navigation

a b s t r a c t

A large percentage of students who use Internet search engines accept the results they are given without challenge, which conflicts with many learning theories that emphasize exploration. Longer and more in-depth searches require critical analyses of results as part of a process that is thought to produce better learning outcomes. With the development of social media, social motivation is increasingly considered an important factor in learning. To boost motivation for knowledge exploration, many instructors are adding elements of high interest to students to their teaching materials and activities. Here we inves-tigate the effects of social motivation-associated priming on learner behaviors involving search engine results, using differences in learner characteristics (peer relationships and math skills) and search question type (add-on and integrated) as the main distinguishing factors. Our sample consisted of 81 students between the ages of 14 and 15 in a junior high school located in northern Taiwan. All of the participants were experienced in using computers, the Web, and search engines. Our results indicate (a) no significant effects of peer relationships or prior math achievement on the use of search results in that subject area; (b) the use of add-on questions (i.e., presenting math questions after presenting inter-personal relationship questions) encourages the appropriate use of summary results; and (c) the use of integrated questions (i.e., learners read stories involving a mix of interpersonal relationship and math information before addressing search questions) encourages students to read more information from search results rather than focusing onfinding single “correct” answers. We interpret the third finding as evidence of greater enjoyment of exploration.

1. Introduction

Search engines are standard features in today’s classrooms, but asMerrill (1997)notes,“information is not instruction,” therefore the information presented by search engines must be modified to have value. Search engines are much better for disseminating facts and ideas than for providing learning materials. Many students never learn how to analyze search results in terms of accuracy and trustworthiness, resulting in lower learning efficiency and occasional feelings of helplessness (Liu, Chin, & Ng, 2003; Spink, Wolfram, Jansen, & Saracevic, 2001).Sparrow, Liu, and Wegner (2011)contend that most search engine users understand more about how tofind information than how to analyze, use, or select appropriate content.

According toBruner (1996), learning by discovery requires students to use inferences, solve problems, andfind facts on their own; the main job of instructors is to encourage curiosity (see alsoDodge, 1995). However, the most popular search engines tend to hinder knowledge exploration by arranging orders of hits based on paid placement or the number of previous clicks on existing links. In a study involving the Google search engine,Lorigo, Hembrooke, Joachims, Granka, and Gay (2006)found that 96% of their participants only looked at thefirst results page, which usually contained ten hits. Most only looked at thefirst two hits, and none went beyond the third page. In other words, they accepted hits at the very top of any search engine return list as containing trustworthy information, and made little (if any) effort to test or challenge them.

For the present study we assume that greater search depth (i.e., longer search times) is an indicator of greater effort to explore infor-mation so as to achieve better learning efﬁciencies. Several research teams have observed that the motivation to use search tools is either

* Corresponding author. Tel.: þ886 3 571 2121x59284. E-mail address:andy910389@gmail.com(S.-H. Ye).

Contents lists available atScienceDirect

Computers & Education

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / c o m p e d u

(2)

extrinsic or intrinsic. Extrinsically motivated students passively search for answers to questions designed by their instructors and focus their efforts onfinding “correct” responses; intrinsically motivated students actively search for information based on personal requirements or interests (Segev & Baram-Tsabari, 2012; Song & Zhang, 2011; Taksa, Spink, & Goldberg, 2008; Thatcher, 2008; Wang, Zhang, & Zhang, 2009). Today’s instructors and knowledge facilitators must deal with challenges associated with a third typedsocial motivation, exemplified by Facebook and other social network websites. Thus, they may benefit from greater understanding of the learning effects of peer interaction in terms of assistance and priming (Mazer, Murphy, & Simonds, 2007). Our primary research focus is the potential effects of peer relationships on learner behaviors tied to search engine-based learning. Second, we will investigate the effects of social motivation, question order (add-on), and situated questions (integrated) on the learning of problem-solving skills involving search engines. For add-on questions, learners respond to search tasks involving interpersonal relationships before performing searches for math-related information. For integrated questions, learners read stories containing a mix of interpersonal relationship and math information before performing searches in response to a math-related prompt.

Our main goal is to understand the combined effects of question design and interpersonal relationship topics on the ways that learners use search engine results in order to enhance learning and information processing. Our primary research questions are

1. Do peer relationships affect learner behaviors involving the use of search engine results?

2. Does previous math achievement affect learner behaviors involving the use of search engine results?

3. In math-related search tasks, can speciﬁc learner behaviors involving search engine results be encouraged by the use of add-on questions directed at interpersonal relationship topics?

4. In math-related search tasks, can speciﬁc learner behaviors involving search engine results be encouraged by the use of integrated questions directed at interpersonal relationship topics?

2. Previous research 2.1. Information searches

Most search engine users, including students, are more interested in_{ﬁnding information in response to speciﬁc problems than general} knowledge accumulation. Today’s students must develop capabilities to search for, process, and evaluate information (Bilal & Kirby, 2002) while integrating prior knowledge (Brand-Gruwel, Wopereis, & Vermetten, 2005). Toward these goals, Web-based information searches are now considered an essential skill for problem-solving (Park & Black, 2007).Kuhlthau (1993)describes Web-based search engine searches as examples of constructive self-knowledge expansion by users dealing with questions or issues, whileMarchionini (1997)views them as responses to learning tasks in which the goal is to modify existing knowledge. They both note that individuals who conduct information searches must establish goals that they can use to evaluate search results, with task completion based on metacognitive judgments. 2.2. Online information search issues

Henry (2006)is among many researchers who have commented on the challenges the Internet presents in terms of reading and in-formation search strategies. According toHölscher and Strube (2000), Internet users who lack experience tend to make decisions regarding objective information too quickly, at the expense of correctness and completeness. Many studentsfind it difficult to master the concept of “keywords,” while others lack the motivation to perform anything more than superficial searches (see alsoBilal, 2000; Hirsh, 1999).Teevan, Alvarado, Ackerman, and Karger (2004)report that American university students in their study were unlikely to systematically analyze search engine information due to their lack of knowledge about organizing concepts during the search process. Younger students can easily get lost if they fail to organize concepts when performing searches, which explains the growing number of studies on Web-based infor-mation integration and application.

Eisenberg and Berkowitz (1990, 2000)“Big Six Skills” approach to using information tools, which is currently popular in computer integrated instruction, emphasizes learning outcome evaluation, assisting the parents of learners in homework direction, and helping teachers learn and implement project design strategies. The six skills are task deﬁnition, information seeking strategies, location and access to information, use of information, synthesis, and evaluation. As part of the Big Six Skills curriculum, the term“use of information” separates reading information (i.e., Web-based information that users browse to perform a task) from summary information (i.e., information that users extract from browsed information to solve a problem) (Donham & Steele, 2008; Eisenberg, 2010; Eisenberg & Berkowitz, 1996; Jansen, 1997; Spitzer, 2000; Wolf, Brush, & Saye, 2003).

2.3. Information-seeking behavior

Kakai, Ikoja-Odongo, and Kigongo-Bukenya (2004)define information-seeking behavior as methods for gathering and sourcing infor-mation for personal use, knowledge updating, and development. For Web-based inforinfor-mation searches it also involves the integration of browsing and search behaviors (Bates, 2002; Choo, Detlor, & Turnbull, 2000). For this study we will define information-seeking behavior as ways that individuals transform thinking into action so as to create paths for fulfilling their information needs.

Accordingly, our focus will be on methods that learners use to establish search engine strategies and to generate usable search results. 2.4. Factors affecting search behaviors

Seven factors that affect Web-based search engine behaviors are information needs, tasks, search systems, knowledgeﬁelds, environ-ment, search results, and obstacles (Marchionini, 1989).White and Iivonen (2001)list of factors includes search task comprehension, task type, Web experience, and background knowledge. After reviewing English-language studies of Web search behaviors published between

(3)

1995 and 2000,Hsieh-Yee (2001)created another list that included examples, information organization and presentation, search task type, Web experience, cognitive abilities, and affective states. However, it is important to note that Hsieh-Yee’s list appeared just before the explosion of social media websitesda reminder that such lists should be viewed as ephemeral.

2.5. Recommended search engine mechanisms

Educators and researchers have modiﬁed search engine sorting algorithms in order to develop multiple search strategies and techniques and to increase retrieval quality. This is especially true for Boolean retrieval and page ranking processes that use retrieval term position and frequency to sort pages in terms of relevance (Sullivan, 2001). According to the Boolean retrieval process, terms that appear in the title tags of web pages have higher correlations than those appearing in titles alone or in main text bodies. Higher frequencies of retrieval terms appearing in web pages mean greater relevance to search goals. Page rankingdespecially Google’s PageRank link analysis algorithmdis based on calculations of mutual hyperlinks among pages. PageRank considers a link from page A to page B as an A-page vote for page B, and determines new page levels in terms of voting sources, including sources of sources (i.e., pages containing links to page A) and the original level of page B. Accordingly, low-level pages can be upgraded via the power of high-level pages.

The main drawback of PageRank is that the value of existing pages is often higher than those of newly created pages due to the system’s dependence on external link numbers rather than content quality (Page, Brin, Motwani, & Winograd, 1999). In response to this and other identified drawbacks, some search engines have modified their page ranking logic to consider factors such as link correlations, stability, and authority, thereby adding value to the PageRank algorithm. In 2010, Google began experimenting with “þ1” buttons next to links in searched pages (a system similar to the Facebook“like” function) that helps users quickly locate content recommended by trusted friends and acquaintances. According to Google, the system reflects both webpage relevance and ranking (Spiro, 2011).

Despite improvements and modifications, these and other mechanisms found in today’s search engines do not encourage extended exploration. A significant percentage of users unwittingly express trust in recommendation mechanisms by accepting the first few search results as containing the answers they need rather than taking the time to look at other options, which frequently results in limited learning (Liu et al., 2003; Spink et al., 2001). Search engine recommendation systems are becoming increasingly sophisticated, yet most users still tend to remember more about how information was found rather than its content. In other words, the Internet has become one of the best examples of what psychologists refer to as“transactive memory” (Sparrow et al., 2011).

2.6. Motivation for using search engines for learning purposes

WebQuest, described as“an inquiry-oriented approach to Web learning,” has been successfully used to help students from grade school to post-secondary institutions navigate through guided learning environments (Dutt-Doner, Wilmer, Stevens, & Hartmann, 2001; Joseph, 2000; Patterson & Pipkin, 2001; Pohan & Mathison, 1998). Its creators believe that Web-based learning should be strongly linked to exploration, as opposed to having learners simply accept the information and links produced by search engine recommendation systems. They also argue that understanding learner motivation for performing searches is an important factor for facilitating greater search depth and encouraging exploration. Two motivation categories have been the focus of multiple studies: intrinsic, referring to activities that provide satisfaction to an individual; and extrinsic, referring to task completion rewards or outcomes (Deci, Koestner, & Ryan, 1998, 1999; Deci & Ryan, 1985). Simple curiosity and“learning for learning’s sake” are examples of intrinsic motivation. In contrast, goal-oriented extrinsic motivation is based on perceptions of rewards in return for task completion. The list of researchers who have addressed self-interest and needs-based issues associated with intrinsic motivation includeSegev and Baram-Tsabari (2012), Serdyukov, Hiemstra, and Ruthven (2010), andSong and Zhang (2011). Those addressing search task or goal-oriented issues tied to extrinsic motivation includeTaksa et al. (2008), Thatcher (2008), andWang et al. (2009).

2.7. Effects of social motivation on web learning

In light of statistics such as one billion Facebook members (Zuckerberg, 2012), it is increasingly important for educators to acknowledge the power of social networks and media tools such as MSN, Facebook and Twitter, and to consider the potential use of such networks in support of learning activities (Mazer et al., 2007). The addition of social tools to some e-learning systems may increase the conﬁdence levels of users during interactions with other members of a team, thereby enhancing overall learning outcomes and encouraging the continued use of a learning system (Liu, Chen, Liu, Lin, & Chan, 2010; Newell, David, & Chand, 2007).

2.8. Priming effect

Priming is deﬁned as new knowledge induced by exposure to information; priming effect (a cognitive psychology term) refers to the processing of a stimulus in response to recent exposure (Yoo, 2008). AsMeyer and Schvaneveldt (1971)observed, the human brain is a knowledge network: when a knowledge node is stimulated, adjacent nodes that are linked to it are also stimulated. Attitude is another factor affecting thoughts and behaviors (Bargh, Chaiken, Govender, & Pratto, 1992; Bargh, Chaiken, Raymond, & Hymes, 1996). In education, teachers can add activities or topics of high student interest to their original instructional materialsdan example of an educational psy-chology priming effect known as the Premack Principle. The same concept is also associated with a brain-compatible teaching approach.

Sylwester (1995)is one of many researchers describing interactions among brains cells (or link nodes, or components of a large nerveﬁber network) during learning. According to this observation, Sylwester offers three lesson and project design suggestions to teachers: add social activities, use a mix of passive (e.g., direct teaching) and active (e.g., group discussion) teaching strategies, and integrate role playing, singing, games, or stories into their lesson plans.

Situated learning theorists describe knowledge as a product of interactions between learners and situations, with knowledge obtained through active inquiry and exploration (Brown, Collins, & Duguid, 1989). Life situations can be viewed as starting points, with learners using practical activities to practice skills and to both absorb and meaningfully interpret knowledge. Stated differently, situations can be used to

(4)

induce, lead, and support learning behaviors, based on the premise that learning is easily generalized and transferred when problems are linked to life experiences.Norman and Schmidt (1992)note that students beneﬁt greatly when they can make connections between learning tasks and their own lives. Accordingly, instructors can beneﬁt from learning and practicing situated question design skills.

Our question design method is based on the classification system proposed byMatthews (1994)for integrating history and philosophy of science principles into science teaching. He identified two types of teaching strategy: integrated, referring to the integration of situated stories and original teaching materials into newly situated teaching materials; and add-on, in which situated stories and teaching materials are distinctly separate (Matthews, 1994; Teixeira, Greca, & Freire, 2012). For the present study we used both types to design search questions involving interpersonal relationship topics to investigate whether question order and/or situated questions encourage learners to use search results in specific ways.

3. Methods 3.1. Research design

As shown inFig. 1, theﬁrst step of our study involves an investigation of the effects of peer relationships and prior math achievement scores on the use of search engine search results. In the second part, we use different question types involving the topic of interpersonal relationships to determine whether question order and/or situated questions encourage speciﬁc ways of using search results (reading and summary) based onEisenberg and Berkowitz’s (1990)Big Six Skills approach. We borrowedLin and Tsai’s (2005)method of using the numbers of browsed and clicked web pages as reading search result indicators, and the number of adopted web pages (i.e., pages containing information used to answer the search question) as an indicator of summary search results. These served as our dependent variables. Our independent variables were learner characteristics (peer relations and math scores) and question type (add-on and integrated).

3.2. Participants

Our initial sample consisted of 92 eighth grade students (54 male, 38 female) between the ages of 14 and 15 in three classes in a junior high school in northern Taiwan. The school’s location and student composition makes it representative of junior high schools throughout Taiwan. Students with learning disabilities were excluded. We chose three classes taught by the same teacher to ensure that the data were not affected by different inﬂuences from multiple teachers. Participants were requested to complete a survey instrument. A total of 81 usable questionnaires were returned by 45 male (55%) and 36 female (44%) studentsdsimilar to the gender distribution for all eighth graders in Taiwan for 2013 (52.3% male, 47.6% female) (MOE, 2013). All of the students had taken computer courses since elementary school, which included training in how to search for information on the Web for class assignments (Tu, Shih, & Tsai, 2008). However, none of them received instruction on how to analyze search results.

Almost all junior high school teachers in Taiwan are encouraged to attend information literacy training courses (Wen & Shih, 2008). The teacher in this study had basic computer and information skills prior to the experiments: general knowledge of how to use computer and information technology (IT) applications; the ability to use computers, networks and media in teaching; the ability to search, assess, organize, and utilize information; and the ability to apply IT tools to problem-solving tasks.

3.3. Data collection

We used six methods to gather and/or record data:

1. An author-designed Internet search engine with data closure and dummy keyword functions (Fig. 2). The data closure feature prevented study participants from ignoring a presented link list and using their own keywords for search tasks. The dummy keyword function ensured that the engine presented the same content regardless of which keywords were used, thereby mitigating the effects of existing learner search skills and search strategies.

(5)

Fig. 3presents an example of search results for the homogenous series search task. Theﬁrst, second, and sixth items were considered “best search results” because they contained clear examples and concept descriptions. The other returned items were not considered useful because they were focused on programming or multiple types of series.

2. A program for measuring both browsing history and duration, which was used to determine learner progress in reading search results.

3. A peer relations questionnaire. Each participant was asked to identify a maximum of_{ﬁve classmates that they got along with and ﬁve} they did not. Responses were used to create groups that served as the basis for our sociometric measure, to be described in a later section.

4. A paper-and-pencil search task sheet whose purpose was to mitigate the effects of prior knowledge. A sequence from the students’ mathematics curriculum served as the topic for our search questions. According toTu et al. (2008), learners with more web experience achieve better search outcomes from closure tasks in which existing answers are constant, while learners with rich content knowledge receive better outcomes from open search tasks in which targets are uncertain and demand non-speciﬁc. For these reasons we used semi-open search questions with content based on constructivist teaching methods, which are thought to arouse curiosity and chal-lenges so as to enhance learning initiative (Dana & Davis, 1993).

5. Math scores. For purposes of statistical analysis, we used the participants’ math scores from the grading period just prior to the experiment to place them in one of three categories: high, medium or low.

6. A modiﬁed version ofLin and Tsai’s (2005)network navigationﬂow chart to record all individual search processes.

Fig. 2. Example of search task interface for peer relationships from the author-designed Internet search engine.

(6)

3.4. Experiment procedure

We used two eighth grade math concepts as searchfield tasks for our experiments: harmonic series and Lucas series. In the two ex-periments conducted for thefirst part of this study, we focused on peer relationships and math achievement. The peer relationship clas-sifications in the first experiment were based on student popularity, measured as social preferences and degree of social impact on peers. Class sections served as units for analyzing differences in ways that learners used search results according to student peer relationships. Students in each class were divided intofive groups based on those relationships (discussed in the following section) before being presented with thefirst search question. In the second experiment, we divided the participants into three groups based on their math scores. Our focus in the two experiments in the second part of the study was on the two search question typesdadd-on for the harmonic series topic, and integrated for the Lucas series. Individual classes again served as our unit of analysis for these two experiments, with students in each class performing search tasks based on the two question types (Table 1).

The browsing history program shown inFig. 4was used to record all search processes, which were later analyzed using network navigation_{ﬂow charts. Last, we used statistical tests to examine how the participants responded to and worked with their individual search} results.

4. Data analysis

We usedCoie and Dodge’s (1983)sociometric nomination measures to determine peer relationships among the study participants. The measures consist ofﬁve sociometric status categories based on each individual’s social preferences: popular, rejected, average, controversial and neglected. Calculations entailed counting and standardizing“liked most” (LM) nominations and subtracting “liked least” (LL) nomi-nations identi_{ﬁed in the questionnaire described in an earlier section.}

For our quantitative analyses we focused on correlations between reading data and summary data for the two learner characteristics, using one-factor ANOVAs and post-hoc tests for separate correlation analyses of the use of search results versus either learner peer re-lationships or learner math scores. Independent sample t-tests were used to determine responses to the third and fourth research questions. The following example from the third experiment (harmonic series search task) shows how peer relationship add-on questions were positioned before the mathematics questions.

Peer relationship questions:

Q1: Please list ten ways in which you could improve your relationships with other classmates.

Q2: Please write three advantages of having good interpersonal relationships, based on your personal experiences. Harmonic series questions:

Q1: Please give a de_{ﬁnition of “harmonic series” based on the information you retrieved from your search.}

Table 1

Experiment details for the two search question types.

Class Question order (add-on) Situated questions (integrated)

1 Interpersonal relationship after harmonic series. Lucas series question plus interpersonal relationship topic. 2 Interpersonal relationship before harmonic series. Lucas series question plus interpersonal relationship topic. 3 Interpersonal relationship before harmonic series. Lucas series question without any other topic.

(7)

Q2: If you were a teacher, how would you introduce harmonic series features and apply them to real-life situations?

The following text from the fourth experiment shows how we used integrated questions in support of the search task designdthat is, how we used a mix of interpersonal relationship and math information to write a story introducing the Lucas series questions.

Do you know why President Obama received the most votes in the American presidential election? Do you know why superstar Michael Jackson was so popular? The primary reason is that both of them understand an important secret about interpersonal relationships. A researcher named Lucas_{ﬁgured out a way to calculate what he called “sequences.” The method made Lucas a famous mathematician, and} his discovery came to be called the“Lucas series.”

If you want to become as popular as Barack Obama and Michael Jackson, you must understand the Lucas series secret. Q1: Please explain the relationship between Lucas and Fibonacci series.

Q2: A Lucas series is a type of prophecy that can bring about good interpersonal relationships. Please write down the Lucas series prophecies that you searched for.

5. Results

5.1. Effects of peer relationships on learner use of search results

As seen in the ANOVA results presented inTable 2, all p values exceeded .05 in the three classes, indicating no statistically signiﬁcant differences in terms of peer relationship type among the study participants.

5.2. Effect of math achievement on the use of search results

According to the one-way ANOVA results for learning mode shown inTable 3, no statistically significant differences were found for numbers of browsed, clicked, or adopted web pages among students with different math aptitude scores who completed the harmonic series search task. Also, no statistically significant differences were found in terms of numbers of browsed or clicked web pages among students with different math aptitudes whofirst completed the interpersonal relationship search task (Table 4).

However, a signi_{ﬁcant difference was found in the number of adopted web pages (F ¼ 4.272; p ¼ .017 < .05), therefore the null hypothesis} is rejected. Post-hoc test results indicate a signiﬁcant difference between high and low math aptitude students regarding the successful execution of the interpersonal relationship search task (p¼ .046 < .05). The positive mean deviation (I J) value indicates that the number of adopted web pages was larger among the higher math aptitude students (M.D.¼ *.04) (Table 5).

Regarding the Lucas series search task, one-way ANOVA results indicate no statistically signiﬁcant differences in the numbers of browsed, clicked, or adopted web pages among students with different math aptitude scores (Table 6).

5.3. Effect of add-on questions on the use of search results

For the third research question, the data shown inTable 7indicate no signi_{ﬁcant differences in terms of the numbers of browsed and} clicked web pages, but a signiﬁcant difference in the number of adopted web pages (t ¼ 2.282; p ¼ .025 < .05). Also, as shown inTable 8 Table 2

ANOVA data for peer relationships and use of search results for the three classes.

Search results Class 1 Class 2 Class 3

F value Significance (p) F value Significance (p) F value Significance (p)

Harmonic sequence Number of browsed web pages .634 .643 1.747 .177 .965 .447

Number of clicked web pages 1.895 .144 1.932 .142 .376 .823

Number of adopted web pages 2.028 .122 2.127 .113 .510 .729

Interpersonal relationships Number of browsed web pages 1.402 .263 .371 .827 2.180 .106

Number of clicked web pages .788 .544 .845 .513 1.734 .180

Number of adopted web pages 1.248 .317 .000 1.000 1.012 .424

Lucas sequence Number of browsed web pages 1.427 .256 .519 .723 .777 .553

Number of clicked web pages 2.317 .086 1.113 .377 .780 .551

Number of adopted web pages 1.264 .312 .338 .849 .159 .957

Table 3

ANOVA data for math achievement, harmonic series questions, and how search results were used.

Search result Sum of squares Degree of freedom Sum of mean squares F Signiﬁcance (p)

Number of browsed web pages Between 1.575 2 .787 1.676 .194

Within 36.647 78 .470

Sum 38.222 80

Number of clicked web pages Between 115.241 2 57.621 2.479 .090

Within 1812.635 78 23.239

Sum 1927.877 80

Number of adopted web pages Between 2.120 2 1.060 1.635 .202

Within 50.571 78 .648

(8)

data for the number of adopted web pages for the harmonic series search task, the number of summary results was clearly higher when study participants were asked interpersonal relationship questions before the math questions.

Combined, the data indicate that (a) summary results were enhanced when the students completed the harmonic series search task after conducting the interpersonal relationship search task, and (b) reading results were not enhanced when they completed the harmonic series search task after the interpersonal relationship search task.

5.4. Effect of integrated questions on the use of search results

For the fourth research question,Table 9data indicate signiﬁcant differences in terms of the numbers of browsed and clicked web pages, but not for the number of adopted web pages. The p and t value data indicate a signiﬁcant effect from the design of situated questions on the number of browsed (t¼ 5.718; p ¼ .000 < .05) and clicked web pages (t ¼ 2.80; p ¼ .009 < .05). Further, reading results for search tasks were higher when prompted by integrated questions.

6. Discussion

Our overall data for thefirst and second experiments indicate no significant differences for the effects of either peer relationships or math aptitude on the ways that the study participants used their search engine results. A possible explanation for the math aptitudefinding is that search habits and methods used by students with different math aptitudes to solve problems using online search tools are similar to the methods they generally use to solve non-math problems. The significant relationship between higher math aptitude and larger number of adopted web pages for interpersonal relationship tasks suggests that students with better math skills prefer a wider variety of information when it comes to addressing interpersonal relationships. One possible reason is that the students with higher math aptitudes had a better grasp of problems, especially in terms of understanding that questions and answers can be more complex than they appear atfirst sight. This result also clearly underscores the relationship between interpersonal factors and question cognition: students with stronger cognition skills are more likely to develop good interpersonal relationships. Since high-aptitude math students are more likely to be asked by their classmates to give assistance with schoolwork, they have greater incentive to try multiple ways to establish and maintain peer relationships.

Table 5

Comparative data for math achievement and number of adopted web pages for interpersonal relationship questions.

(I) Math score (J) Math score Mean deviation (I J) Standard error Signiﬁcance (p) 95% Conﬁdence interval Lower limit Upper limit

Low Medium .14 .170 .708 .57 .29 High .14* .160 .046 .81 .01 Medium Low .14 .170 .708 .29 .57 High .27 .157 .243 .66 .13 High Low .14* .160 .046 .01 .81 Medium .27 .157 .243 .13 .66

Note: I, low level of math achievement; J, medium or high level of math achievement; I–J, mean difference in the number of adopted web pages between the I and J sets. Table 4

ANOVA data for math achievement, interpersonal relationship questions, and how search results were used.

Search result Sum of squares Degree of freedom Sum of mean squares F Signiﬁcance (p)

Number of browsed web pages Between .065 2 .033 .056 .945

Within 45.219 78 .580

Sum 45.284 80

Number of clicked web pages Between 32.721 2 16.321 .623 .539

Within 2049.600 78 26.277

Sum 2082.321 80

Number of adopted web pages Between 2.194 2 1.097 4.272 .017

Within 20.028 78 .257

Sum 22.222 80

Table 6

ANOVA data for math achievement, the Lucas series question, and how search results were used.

Search result Sum of squares Degree of freedom Sum of mean squares F Signiﬁcance

(p)

Number of browsed web pages Between 1.279 2 .640 2.494 .089

Within 20.004 78 .256

Sum 21.284 80

Number of clicked web pages Between 24.994 2 12.497 .309 .735

Within 3158.957 78 40.499

Sum 3183.951 80

Number of adopted web pages Between 1.196 2 .598 1.082 .344

Within 43.125 78 .553

(9)

Regarding add-on versus integrated questions, we found that when participants were asked math questions regarding the harmonic series topic after they were asked questions on interpersonal relationships, the results support Sylwester’s (1995)“brain-compatible” teaching strategydthat is, learning effectiveness is enhanced when social activities are considered and incorporated into curriculum design. Our data suggest that adding the element of peer relationships prior to addressing math questions in the form of add-on questions enhanced the study participants’ use of summary results for the math search tasks, but not the use of reading results. Accordingly, the primary beneﬁt of add-on questions may be the asking of interesting follow-up questions that stimulate analytical thinking while learners are actively using a search engine. However, the data also indicate that the study participants were less likely to browse for more information in response to “theme type” questions.

Regarding integrated questions, our results suggest that integrating interpersonal scenarios into math tasks resulted in increased learner curiosity, initiative, and enthusiasm. In this scenario, the students were more likely to browse and click on a higher number of web pages; however, this did not result in a higher number of adopted web pages. Combined, the data suggest that the main reason for integrating questions on interpersonal relationships into Web-based classroom activities should be to increase learner curiosity so that they search for more information in support of peer interactionsdin other words, to encourage greater search depth via increased information browsing to solve math-related questions.

Theﬁndings have implications in terms of new technologies such as the semantic web, which provides rich information via intelligent computing and information sharing among peers using social networksdfactors that are important in terms of search results. The semantic web uses semantic data and their underlying ontologies to enhance searches. This creates at least one problem: users must understand back-end ontologies and knowledge bases in order to formulate queries and/or interpret search results (Lei, Uren, & Motta, 2006; Tran, Cimiano, Rudolph, & Studer, 2007; Zenz, Zhou, Minack, Siberski, & Nejdl, 2009). This highly subjective requirement can produce inaccu-rate returns. Information provided by social networks tends to be more accuinaccu-rate and strongly correlated with search questions because they are peer-recommended (Bao et al., 2007; Heymann, Koutrika, & Garcia-Molina, 2008). However, this can negatively affect user motivation to look for additional information, based on the assumption that it is not worth the effort. Thus, an important issue is how to exhibit the best characteristics of both the semantic web and social networks. Learning goals must be taken into consideration when designing search engine mechanisms that combine web pages with recommendations for additional information from social network sources.

Table 7

Independent sample t-test data for effect of question order on use of search results.

Search result Levene test of equal variance Average number of equal t-tests

F test Signiﬁcance (p) t value Signiﬁcance (two-tailed) Mean deviation

Harmonic sequence Number of browsed web pages .004 .951 .777 .441 .127

Number of clicked web pages .717 .400 .634 .528 .723

Number of adopted web pages 1.066 .305 2.282 .025 .418

Interpersonal relationships Number of browsed web pages 2.114 .150 .939 .351 .164

Number of clicked web pages 12.026 .001 2.763 .002 3.544

Number of adopted web pages 5.020 .028 2.067 .044 .253

Table 8

Data for effect of question order (add-on) on use of search results.

Search task Search result Question order Count Average

number Standard deviation Standard error of mean Harmonic sequence Number of browsed web pages

Interpersonal relationship question after harmonic series search task

29 2.10 .724 .135

Interpersonal relationship question before harmonic series search task

52 2.23 .675 .094

Number of clicked web pages

29 9.10 5.690 1.057

52 9.83 4.453 .618

Number of adopted web pages

29 1.79 .675 .125

52 2.21 .848 .118

Interpersonal relationships

Number of browsed web pages

29 2.41 .825 .153

52 2.25 .711 .099

Number of clicked web pages

29 12.62 6.293 1.169

52 9.08 3.808 .528

Number of adopted web pages

29 1.69 .541 .101

(10)

7. Conclusion

In this study we investigated two possible inﬂuences on learner use of search engine results: individual characteristics (peer relation-ships and math achievement) and question type (add-on versus integrated). Due to the explosive growth in the number of online social networking websites, peer relationships are becoming an important factor in learner motivation to use online search tools. As many classroom teachers have observed, when search engines are applied to tasks associated with interpersonal relationships, learners tend to enthusiastically spend large amounts of time searching for interesting topics to discuss with friends. Our data indicate that student peer relationships and math achievement had no signiﬁcant effects on search result use behaviorsdthat is, the study participants did not change their learning modes or problem-solving skills involving search engines due to differences in peer relationships or math achievement.

We also found that the students’ use of search results was enhanced when questions were presented in a specific order or in a situated format involving integration with interpersonal relationship topics, with thefirst method promoting learner interest in adopting more search results for answering questions, and the second encouraging greater depth in learner search efforts. We therefore suggest that when designing search engine questions, instructors can benefit from initially presenting questions on peer relationship topics. According to our data, add-on questions are more suitable for search engine tasks with a focus on problem-solving, and integrated questions are more suitable when instructors want their students tofind rich information in order to practice reading skills or increase topic knowledge. Acknowledgments

The authors thank the blind reviewers of this paper for their insightful and constructive comments. They are grateful for helpful input on paper organization from Jon Lindemann. They are also grateful for funding from the Ministry of Science and Technology, Taiwan, grant number: 102-2511-S-009-007-MY3.

References

Bao, S., Xue, G., Wu, X., Yu, Y., Fei, B., & Su, Z. (2007). Optimizing web search using social annotations. In Paper presented at the proceedings of the 16th international conference on World Wide Web.

Bargh, J. A., Chaiken, S., Govender, R., & Pratto, F. (1992). The generality of the automatic attitude activation effect. Journal of Personality and Social Psychology, 62(6), 893. Bargh, J. A., Chaiken, S., Raymond, P., & Hymes, C. (1996). The automatic evaluation effect: unconditional automatic attitude activation with a pronunciation task. Journal of

Experimental Social Psychology, 32(1), 104–128.

Bates, M. J. (2002). Toward an integrated model of information seeking and searching. The New Review of Information Behaviour Research, 3, 1–15.

Bilal, D. (2000). Children’s use of the Yahooligans! Web search engine: I. Cognitive, physical, and affective behaviors on fact-based search tasks. Journal of the American Society for Information Science, 51(7), 646–665.

Bilal, D., & Kirby, J. (2002). Differences and similarities in information seeking: children and adults as web users. Information Processing & Management, 38(5), 649–670. Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem solving by experts and novices: analysis of a complex cognitive skill. Computers in Human

Behavior, 21(3), 487–508.

Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42. Bruner, J. S. (1996). The culture of education. Harvard University Press.

Choo, C. W., Detlor, B., & Turnbull, D. (2000). Information seeking on the web: an integrated model of browsing and searching. First Monday, 5(2). Coie, J. D., & Dodge, K. A. (1983). Continuities and changes in children’s social status: a ﬁve-year longitudinal study. Merrill-Palmer Quarterly, 261–282.

Dana, T., & Davis, N. (1993). On considering constructivism for improving mathematics and science teaching and learning. The Practice of Constructivism in Science Education, 325–333.

Deci, E. L., Koestner, R., & Ryan, R. M. (1998). Extrinsic rewards and intrinsic motivation: Clear and reliable effects. Unpublished manuscript. University of Rochester. Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin,

125(6), 627.

Deci, E. L., & Ryan, R. M. (1985). Self-determination. Wiley Online Library.

Dodge, B. (1995). WebQuests: a technique for internet-based learning. Distance Educator, 1(2), 10–13.

Donham, J., & Steele, M. (2008). Instructional interventions across the inquiry process. College & Undergraduate Libraries, 14(4), 3–18.

Dutt-Doner, K., Wilmer, M., Stevens, C., & Hartmann, L. (2001). Actively engaging learners in interdisciplinary curriculum through the integration of technology. Computers in the Schools, 16(3–4), 151–166.

Eisenberg, M. B. (2010). Information literacy: essential skills for the information age. DESIDOC Journal of Library & Information Technology, 28(2), 39–47. Eisenberg, M. B., & Berkowitz, R. E. (1990). Information problem solving: The big six skills approach to library & information skills instruction. ERIC. Eisenberg, M. B., & Berkowitz, R. E. (1996). Helping with homework: A parent’s guide to information problem-solving.

Eisenberg, M. B., & Berkowitz, R. E. (2000). Teaching information & technology skills: The Big6 [TM] in secondary schools. ERIC.

Henry, L. A. (2006). SEARCHing for an answer: the critical role of new literacies while reading on the Internet. The Reading Teacher, 59(7), 614–627.

Heymann, P., Koutrika, G., & Garcia-Molina, H. (2008). Can social bookmarking improve web search?. In Paper presented at the proceedings of the 2008 international conference on web search and data mining.

Hirsh, S. G. (1999). Children’s relevance criteria and information seeking on electronic resources. Journal of the American Society for Information Science, 50(14), 1265–1283. Hölscher, C., & Strube, G. (2000). Web search behavior of internet experts and newbies. Computer Networks, 33(1), 337–346.

Hsieh-Yee, I. (2001). Research on web search behavior. Library & Information Science Research, 23(2), 167–185.

Jansen, B. A. (Dec 1997). Teaching information problem-solving: the trash-n-treasure note-taking technique. Big6 Newsletter, 1(2), 13. Table 9

Independent sample t-test data for effect of situated questions (integrated ) on use of search results. Search results for Lucas sequence task Levene test of equal

variance

Average number of equal t-tests

F test Signiﬁcance (p) t value Signiﬁcance (two-tailed) Mean deviation

Number of browsed web pages Equal variance assumed 44.773 .000 5.718 .000 .654

Equal variance not assumed 5.718 .000 .654

Number of clicked web pages Equal variance assumed 23.269 .000 2.800 .007 4.038

Equal variance not assumed 2.800 .009 4.038

Number of adopted web pages Equal variance assumed .381 .540 1.858 .069 .385

(11)

Joseph, L. C. (2000). FoodQuest for health. Multimedia Schools, 7(1), 34–37.

Kakai, M., Ikoja-Odongo, R., & Kigongo-Bukenya, I. (2004). A study of the information seeking behavior of undergraduate students of Makerere University, Uganda. World Libraries, 14(1).

Kuhlthau, C. C. (1993). Implementing a process approach to information skills: a study identifying indicators of success in library media programs. School Library Media Quarterly, 22(1), 11–18.

Lei, Y., Uren, V., & Motta, E. (2006). SemSearch: a search engine for the semantic web. Managing knowledge in a world of networks (pp. 238–245). Springer.

Lin, C.-C., & Tsai, C.-C. (2005). Navigationﬂow map method of representing students’ searching strategies on the web. In Paper presented at the world conference on educational multimedia, hypermedia and telecommunications.

Liu, Y., Chen, H., Liu, C., Lin, C., & Chan, H. (2010). A model to evaluate the effectiveness of collaborative online learning teams– self-disclosure and social exchange theory perspective. International Journal of Cyber Society and Education, 3(2), 117–132.

Liu, B., Chin, C. W., & Ng, H. T. (2003). Mining topic-speciﬁc concepts and deﬁnitions on the web. In Paper presented at the proceedings of the 12th international conference on World Wide Web.

Lorigo, L., Pan, B., Hembrooke, H., Joachims, T., Granka, L., & Gay, G. (2006). The inﬂuence of task and gender on search and evaluation behavior using Google. Information Processing & Management, 42(4), 1123–1131.

Marchionini, G. (1989). Information-seeking strategies of novices using a full-text electronic encyclopedia. Journal of the American Society for Information Science, 40(1), 54–66. Marchionini, G. (1997). Information seeking in electronic environments. Cambridge University Press.

Matthews, M. R. (1994). Science teaching: The role of history and philosophy of science. Psychology Press.

Mazer, J. P., Murphy, R. E., & Simonds, C. J. (2007). I’ll see you on “Facebook”: the effects of computer-mediated teacher self-disclosure on student motivation, affective learning, and classroom climate. Communication Education, 56(1), 1–17.

Merrill, M. D. (1997). Instructional strategies that teach. CBT Solutions, 2(6), 1–11.

Meyer, D. E., & Schvaneveldt, R. W. (1971). Facilitation in recognizing pairs of words: evidence of a dependence between retrieval operations. Journal of Experimental Psy-chology, 90(2), 227.

Ministry of Education [MOE]. (2013). Population statistics of school students at all levels in all city and county districts in Taiwan for the school year of 2013. Fromhttps:// stats.moe.gov.tw/ﬁles/gender/103-3.xls.

Newell, S., David, G., & Chand, D. (2007). An analysis of trust among globally distributed work teams in an organizational setting. Knowledge and Process Management, 14(3), 158–168.

Norman, G. R., & Schmidt, H. G. (1992). The psychological basis of problem-based learning: a review of the evidence. Academic Medicine, 67(9), 557–565. Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank citation ranking: bringing order to the web.

Park, Y., & Black, J. B. (2007). Identifying the impact of domain knowledge and cognitive style on web-based information search behavior. Journal of Educational Computing Research, 36(1), 15–37.

Patterson, N., & Pipkin, G. (2001). Guiding readers to new understandings through electronic text. Voices from the Middle, 8(4), 64–66. Pohan, C., & Mathison, C. (1998). WebQuests: the potential of internet-based instruction for global education. Social Studies Review, 37(2), 91–93.

Segev, E., & Baram-Tsabari, A. (2012). Seeking science information online: data mining Google to better understand the roles of the media and the education system. Public Understanding of Science, 21(7), 813–829.

Serdyukov, P., Hiemstra, D., & Ruthven, I. (2010). Towards accessible search systems. In Paper presented at the SIGIR forum.

Song, P., & Zhang, C. (2011). Understanding the role of intrinsic motivations in information technology usage habit: evidence from instant messenger and search engine. International Journal of Organisational Design and Engineering, 1(3), 163–184.

Sparrow, B., Liu, J., & Wegner, D. M. (2011). Google effects on memory: cognitive consequences of having information at ourﬁngertips. Science, 333(6043), 776–778. Spink, A., Wolfram, D., Jansen, M. B., & Saracevic, T. (2001). Searching the web: the public and their queries. Journal of the American Society for Information Science and

Technology, 52(3), 226–234.

Spiro, R. (2011).þ1’s: the right recommendations right when you want themdin your search results. Retrieved 23.09.11, from http://googleblog.blogspot.tw/2011/03/1s-right-recommendations-right-when-you.html.

Spitzer, K. L. (2000). How to use soda pop,“the Blair Witch Project”, and other methods to help students learn to evaluate web information critically. Book Report, 18(4), 21–23. Sullivan, D. (2001). How search engines rank web pages Accessed online at:http://searchenginewatch.com/webmasters/rank.html.

Sylwester, R. (1995). An educator’s guide to the human brain. Alexandria, VA: Association for Supervision and Curriculum Development. Taksa, I., Spink, A., & Goldberg, R. (2008). A task-oriented approach to search engine usability studies. Journal of Software, 3(1), 63–73.

Teevan, J., Alvarado, C., Ackerman, M. S., & Karger, D. R. (2004). The perfect search engine is not enough: a study of orienteering behavior in directed search. In Paper presented at the proceedings of the SIGCHI conference on human factors in computing systems.

Teixeira, E. S., Greca, I. M., & Freire, O., Jr. (2012). The history and philosophy of science in physics teaching: a research synthesis of didactic interventions. Science & Education, 21(6), 771–796.

Thatcher, A. (2008). Web search strategies: the inﬂuence of web experience and task type. Information Processing & Management, 44(3), 1308–1329. Tran, T., Cimiano, P., Rudolph, S., & Studer, R. (2007). Ontology-based interpretation of keywords for semantic search. The semantic web (pp. 523–536). Springer.

Tu, Y.-W., Shih, M., & Tsai, C.-C. (2008). Eighth graders’ web searching strategies and outcomes: the role of task types, web experiences and epistemological beliefs. Computers & Education, 51(3), 1142–1153.

Wang, X., Zhang, C., & Zhang, Z. (2009). Boosted multi-task learning for face veriﬁcation with applications to web image and video search. In Paper presented at the computer vision and pattern recognition, 2009. CVPR 2009. IEEE conference on.

Wen, J. R., & Shih, W. L. (2008). Exploring the information literacy competence standards for elementary and high school teachers. Computers & Education, 50(3), 787–806. White, M. D., & Iivonen, M. (2001). Questions as a factor in web search strategy. Information Processing & Management, 37(5), 721–740.

Wolf, S., Brush, T., & Saye, J. (2003). The big six information skills as a metacognitive scaffold: a case study. School Library Media Research, 6.

Yoo, C. Y. (2008). Unconscious processing of web advertising: effects on implicit memory, attitude toward the brand, and consideration set. Journal of Interactive Marketing, 22(2), 2–18.

Zenz, G., Zhou, X., Minack, E., Siberski, W., & Nejdl, W. (2009). From keywords to semantic queriesdincremental query construction on the semantic web. Web Semantics: Science, Services and Agents on the World Wide Web, 7(3), 166–176.