Pausal behavior of end-users in online searching

(1)

Pausal behavior of end-users in online searching

Mu-hsuan Huang

*

Department of Library and Information Science, National Taiwan University, Taipei 106, Taiwan, ROC Received 6 August 2001; accepted 29 May2002

Abstract

This research used an information processing approach to analyze the pausal behavior of end-users. It is based on viewing the search as a series of actions and pauses (rests). The end-users are 41 students and 3 faculty. After instructions, subjects searched through the semester, doing 79 searches. This study identified reasons for pausing, location of pauses, hesitation rate and pausal behavior changes over time. This study confirms that the searchers pauses less frequentlyand for shorter periods as theyprogressed through searches with more experience and practice, searchers moved more smoothlyonline, and the hesitation rate decreased over time. Over a series of searches or cycles within long searches, searchers gradually began to chunk more information between pauses. However, the duration of pauses do not varysignificantlyover time.

Keywords: Online searching; Information behavior; Information retrieval; Pausal behavior

1. Introduction

During an online search, a searcher alternates between two states: the move or active state, and the pause or rest state. In the move or active state, a searcher issues a command or makes a statement. Conversely, at the pause or rest state, a searcher thinks or does something else instead of moving. An almost imperceptible pause mayindicate that the searcher has anticipated all cognitive decisions or that the decisions require minimal effort. On the other hand, an obvious pause mayreflect the searcherÕs need to concentrate more cognitive effort on making a decision, possibly because of its complexity, novelty, or unexpectedness.

Most online studies have emphasized the move or active states. Yet pausal behavior reflects the cognitive effort involved in decision-making and is an inherentlymore refined approach to understanding information processing in online searchers than error analysis in move states. In a

*

Fax: +886-2-2363-2859.

E-mail address:mhhuang@ccms.ntu.edu.tw(M. Huang).

(2)

preliminary analysis of the data in this study, only about 10% of the pauses were related to errors. Other researchers have studied pausal behavior in silent reading, writing, and problem-solving as a wayof determining decision making (Foulin, 1998; Tavalin, 1995). According to Carter, Ruggels, Jackson, and Heffner (1973, p. 31), readers stopped or paused most frequentlyto think, to reread, to express confusion, to ask an unspecified question, to seek clarification, to make critical comments, and to refer to earlier material. Grunig found that readers stopped when they could not process words into a cognitive structure, when theydisagreed with an evaluation or the evaluative implications of a condition. He later modified his reasons to ‘‘because of confusion, to reread, to ask a question, to think about an idea, because you understand (i.e. to assimilate an idea), to agree, and to disagree’’ (Grunig, 1985).

Since most studies of online behavior have emphasized the active states in a search, little is known about pausal behavior during a search. Consequently, this article emphasizes answering some basic questions about pausal behavior of end-users as theysearch. This research addresses several questions: (1) How frequentlydoes a person pause within a search? (2) For how long does he pause? (3) Whydoes he pause? (4) Does the length of the pause varyaccording to whyhe is pausing? (5) Where does he pause, i.e., during or at the end of a move, and in what moves? (6) What is the size of the chunk of information being processed? (7) Does the pausal behavior change over time as searchers gain experience with searching tasks?

2. Methodology 2.1. Subjects

The studyis based on observing the behavior of end-users over an entire semester. Obtained through convenience sampling, the 44 subjects included 7 (15.9%) undergraduates, 3 (6.8%) faculty, and 34 (77.2%) graduate students (including 1 post-doctorate) in a wide range of disci-plines from Universityof Maryland. Generallyspeaking, subjects came from 16 diﬀerent majors in the University. Among them, 12 (27%) were from history and 10 (23%) were from the De-partment of Agricultural and Resources Economics, only3 (6.8%) subjects came from College of Libraryand Information Services. About 75% of the subjects had used computers before, pri-marilyfor word-processing, although about 23–32% indicated familiaritywith spreadsheets, programming, or another software application. Half had used a CD-Rom search experience before, but only5 (11.3%) had used the Dialog system before. A special search service was es-tablished, and the subjects were allowed to search their own search requests throughout a semester at no charge. Theywere required to use the Dialog search systemÕs command language. Before searching, the seven undergraduates received training in class, the others via a half-hour video-tape. The videotape contained the same materials presented in class training. Also, these two instructions are distributed bythe same person (not the researcher).

2.2. Data

Data consist of 79 diﬀerent searches done bythese searchers, 19 of whom did two to eight searches during the semester. Before each search session, a searcher completed a brief online

(3)

questionnaire to provide information on the natural language query, context of search, previous information gathering, and the subjectÕs assessment of the probabilityof finding information on the topic. In addition, before his first search, each subject completed a questionnaire providing information about personal characteristics, such as academic status and major, and experience with computers, CD-Rom searching, and Dialog searching. In each search session, a subject searched on his own and was observed. The observer made no comments during the search, but a one-page help sheet was always available. The subjects also had access to DialogÕs file-specific help sheets. The subjects did not know their timed keystrokes were captured automatically. The study did not gather thinking-aloud protocols because rambling thoughts would have affected timing, which was critical in this research.

The observer noted pauses and recorded her perceptions of the reasons for them. After the search, while the userÕs output was being printed, the same observer analyzed the keystroke log with a computer program that marked pauses of 3 seconds or longer. Fig. 1 illustrates an example of search commands. It is onlya verysmall part of the subjectÕs search. Extracting from the same search, Fig. 2 shows an example of computer output marked pauses. The same search keys wrap around form the ﬁrst column to the second column. In Fig. 2, the ones marked @@@@ indicate pauses of 3 seconds or longer. As a matter of fact, there are four pauses of 3 seconds or longer in Fig. 2. Then the observer went through the processed keystroke data with the subject to obtain the searcherÕs rationale for each pause of 3 seconds or longer. The researcherÕs observations and the post-search interviews provided corroborative data. Where theydeviated, the interview data were accepted.

In analyzing the data, the emphasis was on the pauses a searcher made while searching. A pause is deﬁned as a discernible stop of 3 or more seconds during the time a person is issuing search com-mands. A total of 2404 pauses 3 seconds or longer were identiﬁed. The studyinvestigated only pauses that occurred while a searcher was issuing commands; it did not cover pauses when the computer was displaying information. Also, this study did not cover system response time and usersÕ reading time. Because the timed keystroke logs reveal only when a user pauses in typing his search statements, theycannot indicate when a user pauses while a computer is displaying information.

3. Analysis and results

Each pause was coded for length of time, the reason for pausing, location in the search bymove (system command) and element (part of multi-part command), and the size of the chunk of in-formation preceding it bynumber of elements. In this study, the searches varied considerably in subject, objective, database, and strategy, since they reﬂected actual information needs. Theyranged from searches emphasizing precision, such as verifying items for a bibliography, to

(4)

retrospective searches for all possible materials on a particular topic. As a result, theyalso differed considerablyin length. Preliminaryanalysis indicated a close relationship between the frequency of pauses and the length of the search. Because of this, the searches were divided into cycles, as in Saracevic, Kantor, Chamis, and Trivison (1988, p. 170), to reduce the effect of search length on other characteristics. A cycle is defined as a cohesive unit of a search, which consists of all con-secutive moves up to, and including one or more print commands. For example, a cycle may start with ‘‘begin’’ or ‘‘select’’ (one ‘‘select’’ command or several ‘‘select’’ commands) and with one or several print commands. Fig. 1 also shows an example of a typical cycle. In this study, the 79 searches in this study yielded 468 cycles.

A search consists of cycles which, in turn, are composed of commands or moves. Commands or moves are composed of elements. During a search, a searcher pauses at various times during the search in connection with processing an amount of information, referred as a chunk of infor-mation. In this study, the aspects of pausal behavior considered are: frequency of pauses, duration of pauses, and reasons for pausing, location of pauses. Also addressed are the amount of infor-mation being processed, hesitation rate, relationship between reasons for pausing and length of pause, hesitation rate, and changes in pausal behavior over time.

3.1. Frequency of pauses

The number of pauses within a search ranged from 1 to 143. On average, a searcher pauses 30.43 times (with a standard deviation of 24.29) in a search. Although some extreme numbers

(5)

existed, most of the searches had 1–30 pauses (61%). Because the frequencyof pause is highly associated with length of a search, breaking a search into cycles helps to reduce the eﬀect of the length of search time. A cycle involved, on average, 5.14 pauses and a standard deviation of 5.6 (67.8%). In general, almost two-thirds of the pauses fell into the range of 0–5 and over 95% of the pauses fell into the range of 0–15 (see Table 1 for detailed information).

3.2. Duration of pauses

Table 2 shows the frequencydistribution of the duration of pauses. The range is large, from 3 seconds to slightlymore than 3 minutes, but the distribution is skewed toward short pauses. Almost 80% were within the range of 3–10 seconds (1920, 79.9%) and 92.6% (2227) of the pauses were within 20 seconds. The average length of a pause was 8.4 seconds and the standard deviation was 10.3 seconds.

3.3. Reasons for pausing

The interview data revealed 250 different reasons for pausing. The subjects phrased their comments veryspecifically, usuallyin terms of the action theywere involved in, and frequently mentioned specific commands, features, or elements. About half the reasons occurred onlyonce or twice (89 times, 36%) (44 times, 18%). Table 3 lists 15 reasons cited most frequentlybythe searchers. The results indicate that the pauses occurred most frequentlybecause subjects were confused about the print format (198 times, 8%) or which set to print (189 times, 8%); wondered which term to use (162 times, 7%); or verified search statements (132 times, 6%). Other pauses occurring frequentlywere caused byconfusion about a database number, a set number previously created (for ‘‘ss’’ commands, not for ‘‘print’’ commands), how to ‘‘display’’ (general syntax problem), how to displaya range of items, which items to print, using the Boolean operator

Table 1

Frequencydistribution of pauses bycycles

Number of pauses per cycle Number of cycles % of total cycles

0–5 317 67.8 0 45 9.6 1 85 18.2 2 67 14.3 3 51 10.9 4 43 9.2 5 26 5.6 6–10 82 17.5 11–15 47 10.0 16–20 8 1.7 21–25 7 1.5 26–30 5 1.1 31–35 2 0.4 Total 468 100.0

(6)

‘‘and’’, where to truncate, and whether to put a space. The subjects also needed to conﬁrm terms, compose search statements, and react to spelling problems.

Because the majorityof online search studies indicate that end-users experienced diﬃculties in developing good search strategies, it was anticipated that most pauses would concern reﬁning a search, such as responding to too manyhits or no postings, deciding to lookup or browse, but the data do not stronglysupport these as reasons for pausing. Instead, most pauses were related to

Table 2

Frequencydistribution of duration of pause

Duration of pauses (s) Number of pauses % of total pauses

3–9.99 1920 79.9 3–3.99 749 31.2 4–4.99 429 17.8 5–5.99 285 11.9 6–6.99 183 7.6 7–7.99 112 4.7 8–8.99 94 3.9 9–9.99 68 2.8 10–10.99 307 12.8 20–20.99 101 4.2 30–30.99 33 1.4 40–40.99 15 0.6 50–50.99 11 0.5 60þ 17 0.7 Total 2404 100.0 Table 3

Most frequentlyoccurring reasons for pausing ranked byfrequency Number

of pauses

% of total pauses

Reasons for pausing

198 8.2 Deciding which print format to use or checking the print format 189 7.9 Determining the set number or checking the set number 162 6.7 Deciding which term to use

132 5.5 Verifying (proof reading) search statements 99 4.1 Checking the set number alreadycreated 93 3.9 Determining how to display(syntax) 76 3.2 Determining which items to print 69 2.9 Determining where to truncate

62 2.6 Deciding the ﬁle number, getting the ﬁle number 48 2.0 Deciding upon spelling (confused about how to spell it) 48 2.0 Composing search statement (formulating search statement) 38 1.6 Determining whether to put a space

34 1.4 Determining whether to use this term (conﬁrming the term)

34 1.4 Determining how to displaya speciﬁc item or a speciﬁc range of items (syntax)

(7)

identifying the set number, determining the print format, deciding which term to use, refining search statements, determining the document number, checking spelling, determining how to display, determining which items to print, and deciding where to truncate. Some pauses confirm the deficiencies and problems of end-users identified in other studies, such as problems in selecting an appropriate database (Kirby& Miller, 1986; Wozny, 1988), selecting terms (Kirby& Miller, 1986; Walker, McKibbon, Hayes, & Ramsden, 1991), using truncation (Charles & Clark, 1990; Trzebiatowske, 1984), choosing print options (Shaw, 1986; Trzebiatowske, 1984; Wozny, 1988), checking spelling errors (Tenopir, 1986), and deciding on Boolean operator (Kirby& Miller, 1986; Tenopir, 1986; Trzebiatowske, 1984; Wozny, 1988).

3.4. Location of pauses 3.4.1. By element

In preliminaryconsiderations of where pausing occurred, the researcher had thought that, because most commands, especiallythose of novice searchers, are short and relativelyuncom-plicated, a command would serve as the coherent unit for the searcher. Considering a word as an integrated chunk of letters in itself, a command rarelyhas more than seven chunks. Research has shown that human short-term memories have the capacityfor 7 (plus or minus 2) chunks (Miller, 1956). As a result, pauses would occur at the end of a search move or coherent thought, i.e. at the end of the command. In this study, a move was deﬁned before as a system command inclusive of its variable parts. ‘‘ss boats or ships,’’ for example, was considered one move, itself composed of some intermediate chunks. Relating pauses to the commands in which theyoccurred would then give some indications of the tasks which required greater information-processing or were less programmable from the searcherÕs point of view.

It is apparent that searchers paused within commands, even, in some cases, within words. As a result, it was necessaryto use a ﬁner unit of analysis than the move. In selecting the unit, the premise was still to retain a sense of the intermediate chunks. Instead of keystrokes, used in many other studies, the smallest unit of analysis for studies of location adopted in this study is an element, as deﬁned earlier.

Considering this unit, most searchers paused when theywere issuing ‘‘ss’’ commands, selecting terms, determining the set number (for both ‘‘ss’’ and ‘‘print’’), solving logical problems (espe-ciallywhen to use ‘‘and’’ and parentheses), and determining print format and item number for print commands. These elements accounted for 75.7% (1821) pauses (Table 4).

Byanalyzing the subjectsÕ comments and noting the location of the element referred to in the comment, it is possible to determine if subjects paused to consider a future action, to assess a previous action, or to think about something theywere doing at the time. Most of the pauses (67% (1615)) happened as subjects were issuing the element; 23% (551) of them occurred after searchers ﬁnished issuing the element; and only10% (238) anticipated an action.

3.4.2. By moves or commands

Some elements occur in several commands and some commands incorporate manyelements so that the problematic nature of speciﬁc commands is obscured somewhat byreporting results only at the element level. Table 5 shows the pause analysis by command. As expected, ‘‘select steps’’ or ‘‘select’’ statements accounted for the majorityof the pauses (1273 times, 53%). Also, ‘‘display’’

(8)

Table 4

Frequencydistribution of pauses byelements

Number of pauses % of total pauses Element

37 1.54 Begin (command part)

75 3.12 Display(command part)

21 0.87 DisplaySet (command part)

2 0.08 Execute Steps (command part) 4 0.17 IdentifyDuplication (command part)

25 1.04 Logoﬀ (command part)

5 0.21 Save or Save Temp (command part) 175 7.28 Select Step or Select (command part)

52 2.16 Type (command part)

415 17.26 Terms 7 0.29 AU (field delimiter) 4 0.17 PY (field delimiter) 6 0.25 HP (field delimiter) 1 0.04 LA (field delimiter) 1 0.04 AN (field delimiter) 4 0.17 CO (field delimiter) 2 0.08 DE (field delimiter) 8 0.33 TI (field delimiter) 1 0.04 SU (field delimiter)

50 2.08 Values of preﬁx part (such as accession number)

115 4.78 Sets alreadycreated (for ss)

52 2.16 Database number

4 0.17 Other variables parts

113 4.70 AND 33 1.37 OR 3 0.12 NOT 120 4.99 (,or) [parentheses] 59 2.45 Truncation (?) 2 0.08 W (proximityoperator) 5 0.21 N (proximityoperator)

6 0.25 Number of words to allow with proximity operator 175 7.28 Enter 24 1.00 , (punctuation) 2 0.08 >(punctuation) 15 0.62 ¼ (punctuation) 142 5.91 / (punctuation) 14 0.58 - (punctuation) 2 0.08 . (punctuation)

289 12.02 Set numbers (for display)

170 7.07 Print numbers 107 4.45 Document numbers 35 1.46 All 1 0.04 TI (print format) 21 0.87 Null elements 2404 100 Total

(9)

accounted for 24% of pauses, and ‘‘type’’ for 14% of pauses. Together these three moves, plus ‘‘begin’’ which selects a database, accounted for 96% of all pauses.

The select and type/display commands are also the most ubiquitous in any search because of their inherent ﬂexibilityand centralityto the search task. We maysuspect that the majorityof pauses occurred in those moves because most subjects used onlya small number of the available commands. Table 6 shows the command analysis. As expected, ‘‘select steps’’ or ‘‘select’’ ac-counted for the majorityof commands (877 times, 40%). Also, the ‘‘display’’ acac-counted for 26.7%, and ‘‘type’’ for 15.1%. The above three moves, plus ‘‘begin’’ which select a database, accounted for 88.7% of all moves. What is also apparent from this table is that the subjects used relativelyfew commands in their searches. Excluding ‘‘displaysets’’ (148 times, 6.7%) and ‘‘logoﬀ’’ (80 times, 3.6%), the other six commands accounted for only0.9% of all commands. It can be concluded that

Table 5

Frequencydistribution of pauses bymoves

Number of pauses % of total pauses Move

114 4.74 Begin

1273 52.95 Select steps or select

567 23.59 Display

333 13.85 Type

1 0.04 Expand

40 1.66 Displaysets

5 0.21 Execute step

6 0.25 Identifyduplication and remove duplication

6 0.25 Save and save temp

8 0.33 Sort

26 1.08 Logoﬀ

25 1.04 Null command

2404 100 Total

Table 6

Frequencydistribution of moves and pauses

Move FrequencyPause Pause/frequency

Begin 151 114 75.5%

Select steps or select 877 1273 145.2%

Display586 567 96.8% Type 332 333 100.3% Expand 2 1 50.0% Displaysets 148 40 27.0% Execute stop 5 5 100.0% Identifyduplication or remove duplication 4 6 150.0%

Save or save temp 4 6 150.0%

Sort 5 8 160.0%

Logoﬀ 80 26 32.5%

(10)

a small proportion of specific commands accounted for a large proportion of the command used. The results confirm FenichelÕs (1981) findings.

Table 6 also shows that, in everymove, searchers pause 1.08 times. People pause often in ‘‘select’’ command and some infrequentlyused commands, such as ‘‘save’’ and ‘‘sort’’ commands. While issuing one ‘‘select’’ command, searchers paused 1.5 times. The result conﬁrms the diﬃ-culties in term selections and term operator selections for novice end-users. Selecting appropriate terms always requires good search strategies and search knowledge. No matter how familiar a searcher is with the search system, he still pauses in selecting terms. Among the other frequently used command, ‘‘display’’ and ‘‘type’’ accounted for 37% pauses. When issuing one ‘‘display’’ or one ‘‘type’’ command, searchers pause almost once. The result can be explained by the complexity of print command. It requires more information processing capacityespeciallyfor novice searchers. To correctlyuse the commands to displayand type citation retrieved, a searcher needed to know the set number, print format, document number and the sequence in which to arrange these for a meaningful command that would have been accepted bythe system.

3.5. Amount of information being processed

A pause is a discernible break in activityand presumablyis related in some wayto the activities that precede it. It can be regarded then as the final step in a series of steps that, for some reason or another, ‘‘hang together’’ for the searcher, i.e. theyare unified and cling together. A chunk is defined as a pause and the moves or elements preceding it until the previous pause. The size of a chunk can be viewed as an indicator of information processing capacityof a subject. Drawing on data about the frequencyof pauses, the average cycle, then, would have included five chunks of information, but mayhave included as few as none and as manyas 34.

The size of the chunk was measured bythe number of elements. In this study, an element is any component of a command and its variable part, such as terms. It mayconsist of one or several keystrokes but the keystrokes are related. It is akin to a morpheme. For example, ‘‘d sl/5/all’’ consists of six elements (d,sl,/,5,/,all) and ‘‘ss expansion?’’ consists of three elements (ss, expan-sion, ?). Elements were used instead of commands because the subject frequentlypaused within commands, sometimes several times within the same command. Theysegmented the commands. Studies of problem-solving have generallyindicated that, although both novices and experts chunk information, and the number of chunks that can be processed is relativelyinvariant, the size of the chunks varies with the level of expertise of the problem-solver. Experts tend to integrate more information into a chunk than novices (Newell & Simon, 1972). As expected, because most subjects were novice searchers, the size of most chunks was quite small. A chunk consisted, on the average, of six elements (the standard deviation is 8.9) (see Table 7). Almost 85% of the chunks contained less than 10 elements and lasted less than 30 seconds. A few big chunks existed; for example, 30 chunks had more than 40 elements (maximum: 118).

3.6. Hesitation rate

The hesitation rate, or ratio between pausing and issuing commands (total pausing time divided bytotal user input time), indicates how tentativelya person moves online. The higher the ratio, the more time is spent in pausing. A hesitation ratio of 0.5, for example, indicates that a searcher

(11)

spends equal time issuing commands and pausing between them. In this study, the hesitation rate per search ranged from 0.22 to 0.94. The average hesitation rate per search was 0.68 while the standard deviation is 0.14. On the average, the searchers in this studyspent about 70% of their time online pausing instead of acting. More than half of the searches had hesitation rate between 0.6 and 0.8 (40 searches, 51%). There were some extreme values: two were less than 0.4 (3%) and four were greater than 0.9 (5%). Again, the hesitation rate diﬀers greatlyacross cycles in this study, although in over 75% (78.2%) of the cycles the searchers spent more time pausing than they do issuing commands (see Table 8), sometimes as much as 90%. The average hesitation rate per cycle is 0.63 and the standard deviation is 0.19. Although the distribution was scattered, the majority(74%) fell within the range of 0.5–0.9.

3.7. Relationship between reasons for pausing and length of pause

It is evident that composing search statements and choosing appropriate search terms are complicated tasks that mayrequire longer processing time than determining the set number or deciding whether to use ‘‘sl’’ or ‘‘l’’. The more complicated the decision, the longer the pro-cessing time is likelyto be. Table 9 shows the average duration of the most frequentlyoccurring pauses. The data show that subjects paused longer (greater than 8.8 seconds) when composing search statements, determining how to display, and confirming database numbers. Also, searchers had shorter pauses when determining the set number or item number for printing, deciding where to truncate, deciding upon spelling, and determining whether to space or not (these pauses averaged less than 6 seconds). Most findings confirm that the tasks that required more intellectual effort needed a longer time to process. Composing a search statement had the longest duration of pause, and checking the set number for ‘‘print’’ command and deciding which items to print had the shortest.

Table 7

Frequencydistribution of size of chunk measured bynumber of elements

Size of chunk (Elements) Number of chunks % of total chunks

1–5 1629 67.8 1 597 24.8 2 419 17.4 3 285 11.9 4 180 7.5 5 148 6.2 6–10 394 16.4 11–15 155 6.4 16–20 78 3.2 21–25 63 2.6 26–30 30 1.2 31–35 17 0.7 36–40 8 0.3 41þ 30 1.2 Total 2404 100.0

(12)

3.8. Changes over time

This studytests the general hypothesis that pausal behavior changes over time as searchers gain experience with searching tasks. The independent variable, search experience, is operationalized as the order of the search or the order of the cycle within a search. More specific hypotheses related to the characteristics of pausal behavior in this studyare: as searchers progress through searches or cycles in a search, they pause less frequently and for shorter periods of time. Because searchers with more experience will move more smoothlyonline, so the hesitation rate per search (per cycle) will decrease over time. Over a series of searches or cycles, the chunks of information between pauses will increase. With more experience, end-users will become more fluent in formulating their search, as defined bythe hesitation rate and smoothness of the interaction.

Table 8

Frequencydistribution of hesitation rates bycycles

Hesitation rate per cycle Number of cycles Percentage

0.0–0.09 4 0.9 0.1–0.19 6 1.3 0.2–0.29 17 3.6 0.3–0.39 29 6.2 0.4–0.49 46 9.8 0.5–0.59 82 17.5 0.6–0.69 105 22.4 0.7–0.79 89 19.0 0.8–0.89 70 15.0 0.9þ 20 4.3 Total 468 100.0 Table 9

Most frequentlyoccurring reasons for pausing ranked byaverage duration of pauses Duration of pause (s) Most frequentlyoccurring reasons for pausing Mean S.D

5.0 3.1 Determining the set number or checking the set number (for ‘‘print’’ command) 5.0 2.3 Determining which items to print

5.9 4.1 Determining where to truncate 5.9 3.6 Determining whether to put a space 6.0 3.8 Deciding upon spelling

6.4 5.5 Verifying (proof reading) search statements

6.4 4.8 Checking the set number alreadycreated (for ‘‘ss’’ commands) 6.9 5.1 Determining whether to use this term (conﬁrming the term)

7.2 5.7 Determining how to displaya speciﬁc item or a speciﬁc range of items (syntax) 7.5 7.5 Deciding which term to use

8.0 8.6 Deciding to use ‘‘and’’, making sure to use ‘‘and’’, or wondering whether to use ‘‘and’’ 8.4 8.2 Deciding which print format to use or checking the print format

8.8 9.5 Deciding the ﬁle number, getting the ﬁle number 9.0 8.1 Determining how to display(syntax)

(13)

Table 10 compares the first search with later searches. According to t-test results, all hypotheses are fullysupported at least the 0.01 level. Table 10 shows subjects paused on the average about 36 times in their first searches. Each pause lasted about 9 seconds. The hesitation rate was 0.73, which was slightlyhigher than the average hesitation rate of all searches (0.68). Between each pause the searcher issued an average of six elements. In later searches, the number of pauses dropped byalmost a third (23). The duration of the pause was less but not markedlyso, about 7.5 seconds instead of the 9 seconds in the first search. The hesitation rate declined byabout 16%. The searchers chunked larger amounts of information between pauses (about eight elements instead of six). The results confirm YuanÕs (1997) findings that search experience affected several aspects of end-users behavior, including the increase of search speeds.

Table 11 compares the first cycle with later cycles. According to ANOVA test results, all hy-potheses are fullysupported at least the 0.001 level with the exception of the hypothesis related to length of pauses. The data across the six cycles show that improvement began to occur even in the first few cycles. From learning theory, the most common finding in instrumental learning is that the response builds up rapidlyand then levels off (Houston, 1981). According to the phenomenon, improvement is more apparent in the earlystages. In the first cycle, the subjects paused on the average about 10 times during a cycle. Each pause lasted about 9 seconds. The hesitation rate was 0.71. Between each pause the searchers issued an average of 4 elements. In the sixth or later cycles, the number of pauses dropped byalmost two-thirds to 3 pauses during a cycle. The duration of pause was slightlyless, about 8.26 instead of 9 seconds. The hesitation rate declined byabout 20%. Comparing the first cycle with the sixth cycle, the searchers chunked larger amounts of in-formation between pauses, measured in number of elements (about nine elements instead of four). In the first cycle, for example, a print command of ‘‘t/s5/9/all’’ may have been chunked as ‘‘t- -s5--/- -9- -s5--/- -all’’ (worst case, six chunks, one element per chunk). After one or two cycles, it may have been chunked as ‘‘t s5/- -9/all’’ (two chunks, each chunk with more elements). Fig. 3 shows two consecutive cycles in the same search with pauses noted.

An interesting phenomenon is appeared in Table 11. The size of a chunk and the hesitation rate reached maximum capacity in the fourth cycle. By the ﬁfth cycle, the size of the chunk actually declined, and the hesitation rate stabilized. The notion of cognitive load oﬀers an explanation for this apparent stability. Cognitive overload is likely to happen in a person with less experience. In the beginning searches or cycles in this study, operating basic search commands may have

Table 10

Average frequencyof pause, average duration of pause, average hesitation rate per search, and average size of chunk preceding pause for ﬁrst search and later searches

Search characteristics Searches

First search Later searches Average number of pauses per search _{36.32 (27.18)} _{23.03 (17.82)}

Average seconds per pause in a search _{8.92 (11.12)} _{7.40 (8.27)}

Hesitation rate per search _{0.73 (0.13)} _{0.61 (0.13)}

Average number of elements per chunk in a search _{5.57 (7.85)} _{7.58 (10.65)}

Notes: 1. The ones marked with in the table are signiﬁcant at the 0.01 level. 2. The ones marked with are signiﬁcant at the 0.001 level. 3. The standard deviation for each variable is indicated in parentheses below each average value.

(14)

occupied a large amount of information processing capacity, so less capacity was available for other competing commands of the search system. As searchers became more familiar with the basic search commands, theyfreed capacityto explore other commands or features. But, as the searchers experimented with new commands or features, more information processing capacity again had to be focused on the new features, and theywere not able to chunk as much infor-mation as before. This result also reflects an inverted-U effect as manyphenomena, such as ac-tivation level and performance efficiency(Hebb, 1955), conflict level and the number of new actions and the improvement of accuracyof the interpretation (Chalidabhongse, 2000). The in-verted-U curve also describes the pausal behavior, i.e. learnersÕ success rate actuallygoes down as theymake adjustment to analyzing problem using new paradigm, then goes up when this para-digm mastered, goes down again when theyexplores other features of the system.

Table 11

Average frequencyof pause, average duration of pause, average hesitation rate per cycle, and average size of chunk preceding pause for ﬁrst cycle through sixth and later cycles

Cycles

1st 2nd 3rd 4th 5th 6th

Average number of pauses per cycle

9.84 (8.17) 5.09 (5.53) 5.57 (5.28) 4.37 (4.39) 4.05 (3.85) 3.20 (3.10) Average second per pause in a

cycle

8.97 (11.36) 8.63 (9.90) 7.63 (6.93) 8.38 (9.60) 7.46 (8.58) 8.26 (11.46) Hesitation rate per cycle _{0.71 (0.16)} _{0.67 (0.18)} _{0.64 (0.19)} _{0.58 (0.17)} _{0.58 (0.16)} _{0.57 (0.19)}

Average number of elements per chunk in a cycle

4.05 (5.69) 5.56 (6.46) 5.75 (7.61) 9.19 (11.47) 7.68 (8.20) 8.62 (11.69)

Notes: 1. The ones marked with are signiﬁcant at the 0.001 level. 2. The standard deviation for each variable is indicated in parentheses below each average value.

(15)

The length of pause does not varysigniﬁcantlyacross cycles, indicating perhaps that, for some pauses, task-related factors maycontribute to length diﬀerences within a search. The same task mayrequire similar time regardless of where it appears in a search. In a studyof pausing during a writing task, however, pause location was found to be relative weak predictor of pause duration variations (Foulin, 1998).

In this study, the percentage distribution of types of pause across early and later searches were quite similar. For both first searches and later searches, almost 35% of all pauses related to display and type, 15% to term selection, and 13% to term operators. Table 12 shows the percentage distributions of type of pause among different cycles. In the first cycle, the subjects paused more frequentlyin problems related to selecting databases, issuing ss commands, printing records, and choosing term operators. After several cycles, pauses moved to problems relating to choos-ing terms, refinchoos-ing search statements, dochoos-ing other field searches, and ‘‘movchoos-ing-forward’’ with a search. Although the results maybe due to the natural progression of a search, theyshow more specifically that, in the first few cycles (especially the first), searchers paused more about syntax

Table 12

Frequency distribution of types of pauses for ﬁrst cycle through sixth and later cycles Cycles

1st 2nd 3rd 4th 5th 6thþ Total (a) Pauses related to

‘‘begin’’ a database 50 (6.4%) 9 (2.5%) 14 (3.9%) 7 (3.0%) 5 (3.0%) 10 (2.0%) 95 (4.0%) (b) Pauses related to ss command 79 (10.1%) 27 (7.5%) 27 (7.6%) 17 (7.4%) 15 (9.0%) 41 (8.0%) 206 (8.6%) (c) Pause related to choosing terms 76 (9.8%) 74 (20.6%) 69 (19.3%) 60 (26.0%) 35 (21.0%) 74 (14.5%) 388 (16.1%) (d) Pauses related to ‘‘display’’ or ‘‘type’’ 306 (39.3%) 147 (40.8%) 112 (31.4%) 65 (28.1%) 58 (34.7%) 167 (32.7%) 855 (35.6%) (e) Pauses related to

using term operators

109 (14.0%) 47 (13.1%) 46 (12.9%) 27 (11.7%) 17 (10.2%) 65 (12.7%) 311 (12.9%) (f) Pauses related to reﬁning a search 18 (2.3%) 7 (1.9%) 8 (2.2%) 9 (3.9%) 9 (5.4%) 9 (1.8%) 60 (2.5%) (g) Pauses related to ‘‘ﬁeld search’’ 62 (8.0%) 21 (5.8%) 44 (12.3%) 15 (6.5%) 11 (6.6%) 53 (10.4%) 206 (8.6%) (h) Pauses related to getting help 22 (2.8%) 8 (2.2%) 10 (2.8%) 4 (1.7%) 5 (3.0%) 29 (5.7%) 78 (3.2%) (i) Pauses related to

‘‘moving forward’’ with a search 54 (6.9%) 19 (5.3%) 25 (7.0%) 22 (9.5%) 9 (5.4%) 53 (10.4%) 182 (7.6%) (j) Pauses related to ‘‘logoﬀ’’ 3 (0.4%) 1 (0.3%) 2 (0.6%) 5 (2.2%) 3 (1.8%) 9 (1.8%) 23 (1.0%) Total 779 (32.4%) 360 (15.0%) 357 (14.9%) 231 (9.6%) 167 (6.9%) 510 (21.2%) 2404 (100%) Notes: 1. Types of pauses marked with in the table are signiﬁcant at the 0.05 level, the one with at the 0.01 level, and the one with at the 0.001 level. 2. The percentage for each value is indicated in parentheses.

(16)

decisions. As their experience increased, their abilitywith the syntax, especiallythe basic com-mands and command formats, grew, and theycould allocate more cognitive capacityto the more variable aspects of the messages, such as the search terms. Also, bythe fourth cycle, the searchers seemed to have become suﬃcientlyfamiliar with basic search elements to explore other system features. Bythat stage fewer pauses were related to displayand type, but more were related to selecting terms and reﬁning search statements.

4. Discussion

Searching for information online is a complex task, especiallyfor novice searchers. Besides the normal cognitive demands of formulating a strategyand expressing in search terms combined with Boolean logic, theyalso must cope with the syntax of a foreign language. The latter entails not onlythe specific search terminologyand functions but also the arrangement and punctuation of the search statement. Perhaps the most significant finding in this studyis that searchers became more fluent relatively quickly. Change was obvious even across cycles within the same search, as Fig. 3 indicates. Nevertheless, on average, over half of their time was spent in pausing, although their hesitation rates declined as theygained experience.

Some of this increase in ﬂuencywas due, no doubt, to the fact that the subjects conﬁned themselves to relativelyfew, albeit powerful, command (see Table 6). Even the few theyused cause them to pause while using them. For example, as the element analysis indicated, some subjects were confused about ‘‘ss’’ commands, even about 50% of the searchers had searched before on CD-Roms; manyof the CD-Rom searching have not to remember which command allowed him to develop a set, he also had to reject a familiar tendencyand actuallyuse a command for this function.

That subjects paused often as theywere selecting terms (1273 times, 53%) is not surprising. The choice of commands maybe fairlylimited and decisions regarding them are likely, in the long run, to become somewhat familiar, but search terms change according to the situation in very significant ways, probably always necessitating pauses during the task. Selecting appro-priate terms requires subject knowledge, the abilityto think flexiblyabout term relationships, and understanding the implications of variant ways of phrasing the terms. Even experienced searchers are likelyto pause while selecting terms. In addition, pauses relating to composing a search statement, including selection of search terms were among the longest pauses seen in the study(Table 9). The relativelyinvariant length of time associated with reasons for pausing suggests that task-related factors maybe more influential than experience in explaining pause length.

The adaptabilityof the searchers to the cognitive loads theyfaced showed in several instances, most notably in connection with the ‘‘display’’ and ‘‘type’’. In Dialog, the two are complex commands. At least a quarter of the pauses were related to ‘‘display’’ or ‘‘type’’ (900 pauses, 37.4%). To correctly use the search system commands to display or type citations retrieved, a searcher needed to know the set number, print format, document number and the sequence in which to arrange these, and the punctuation. These elements constitute a large cognitive load for novice users. In addition, for both set and items, theyeither had to note the relevant set numbers or item numbers during the searches or check back to verifythe information. Even within the

(17)

syntax, the range of options is large. In this study, some subjects lessened the cognitive load by eliminating some of the decisions associated with printing or typing. They discovered that, by using ‘‘t’’ or ‘‘d’’ for the commands ‘‘type’’ or ‘‘display’’, they could see one citation at a time. In-stead of programming beyond the default options, they simply continued to type sequential dÕs or tÕs to see the desired citations.

Previous studies of end-user searching behavior based on error analysis have indicated that searchers have diﬃcultywith Boolean logic (Borgman, 1984). Although this studydoes not an-alyze search transcripts and search results to determine the level of the searchÕs success, the subjectsÕ pausing behavior conﬁrms that the subjects paused frequentlyin connection with search logic (including using Boolean operators, the proximityoperator, and parentheses). Again some searchers mayhave been adjusting mental models developed through use of other systems. Most of the subjects indicated some experience with the local OPAC. At Maryland, the OPAC inter-prets words entered without logical operators as ‘‘anded’’ terms.

The Dialog system is not designed for end-users, even though end-users can still learn to use it. The commands used in Dialog system are quite diﬀerent compared to CD-Rom systems or net-work environment. Some pauses identiﬁed from this studyare independent of search systems. Those pauses that are not related to system can arise in other system, such as determining where to truncate, deciding upon spelling, determining whether to use this term or not, etc. But, some pauses that are related to Dialog syntax will not occur in other search systems, such as those pauses related to type and display commands.

Another especiallyimportant ﬁnding is that the subjects processed relativelysmall chunks of information, although size of chunks would lengthen over cycles. They paused often within commands and tended to pause as theyare issuing an element, not to think back, or to plan ahead several moves. The latter suggested that one potential use of monitoring pausing data is to identify places where help could be suggested automatically. A longer pause, for example, might trigger advice about the likelydecisions at that point.

While some of the findings corroborate and support what have alreadybeen discovered about novice searchers using command systems, nevertheless, this study adds a level of detail and re-finement to previous findings. The findings here, for example, can reach the level of identifying potentiallyproblematic areas within a particular command or show how quicklynovice searchers inculcate basic syntax into their searching patterns.

In web-based environment, there are fewer and fewer command-based information retrieval systems. Thus, it becomes the major limitation of this article. However, most ﬁndings can still help the Dialog to improve its command-based system. Also, ﬁndings are independent of command system can be applied to general IR systems. More important, pausal behavior can always identify the potential problem areas of users, especiallyfor some decisions requiring more information processing capacityeven in web-based IR systems.

5. Usefulness of studying pausal behavior

This studymarks the ﬁrst time that pausal behavior has been emphasized in studying online searching behavior, although time-related behavior in the form of response times has been studied in connection with other computer tasks.

(18)

Pauses are sensitive indicators of task areas, subtasks, or command which require greater in-formation processing. Theymaydenote complex or unanticipated demands on the infor-mation processing capacityof the individual involved in a task. A person maypause, for example, to recall information, to react to unexpected output, to conﬁrm an action, to consider alter-native actions and plan, to retrieve information from external sources, or to assimilate infor-mation.

Although some pauses are associated with errors, the pause is far more ubiquitous. As a result, analyzing pausal behavior provides insights not only into decision-making resulting in actual errors, but also into decisions which are more complex or which result in information which is unexpected. It mayprovide the basis, therefore, for identifying potential problem areas. In ad-dition, the approach oﬀers promise for studying information processing capacity which other methods cannot provide. In this study, for example, it was possible, through looking at pauses, to characterize the length and size of chunks. Pausing time can be preciselycalculated from timed keystroke data. Since human-interaction labs regularly collect keystroke data, they should con-sider augmenting this data by analyzing timed keystroke data to identify pauses both between and within commands. Preciselymeasured pauses, used in conjunction with error analysis, for ex-ample, maybe helpful in distinguishing between errors and slips (Norman, 1981).

Unfortunatelykeystroke capture programs cannot note pauses during all subtasks within a search. Theycannot, for example, register the pauses that occur as an individual reads the system output, but theycan readilyidentifypauses when a searcher is inputting commands, even those of relativelyshort duration (see Fig. 2). As this studyhas shown, however, the reason for pausing cannot always be inferred from observations, nor can the person performing the task always remember the reason or reasons for pausing even when given the context for the pause shortly after completing the task. But, in this study, after searchers saw the computer output, they could recall their reasons for pausing. Nevertheless, these problems do not erase the potential beneﬁts of gathering data on pauses within a task.

Determining the size of the pause to be studied in characterizing pausal behavior is not simple since individual differences mayfigure prominentlyin the length of a pause. In a studyof more experienced online searchers, for example, limiting relevant pauses to those three seconds or longer, as in this study, may greatly underrepresent the actual pausing that occurs. Ideally, a pause should be identified as anysignificant change in a search rhythm for an individual. Con-tinued studyof pauses should provide additional insights into its usefulness for understanding the information processing occurring during computer-related tasks.

Pausal behavior provides insight into an individualÕs information processing as he proceeds through a particular task. In a computer environment, much emphasis has been placed on ana-lyzing errors or patterns of commands as a means of gaining insights into the mental models individuals operate under in doing a task (Borgman, 1984; Norman, 1983). Pausal analysis provides a basis for additional insights into human behavior.

Acknowledgement

The author acknowledges the inspiration, assistance, and the contribution of Marilyn D. White to the development of this research.

(19)

References

Borgman, C. L. (1984). The user’s mental model of an information retrieval system: Eﬀects on performance. Ph.D. dissertation, Stanford University.

Carter, R. F., Ruggels, W. L., Jackson, K. M., & Heﬀner, M. B. (1973). Application of signaled stopping technique to communication research. In P. Clarke (Ed.), New models for mass communication research (pp. 15–43). BeverlyHill, California: Sage Publication.

Chalidabhongse, J. (2000). Modeling complexityin organizational dynamics: Organizational learning and conﬂict. Dissertation Abstracts International: Section B: The Sciences and Engineering, 61(1-B), 463.

Charles, K. S., & Clark, K. E. (1990). Enhancing CD-ROM searches with online updates: An examination of end-user needs, strategies, and problems. College and Research Libraries, 51, 321–328.

Fenichel, C. H. (1981). Online searching measures that discriminate among users with diﬀerent types of experiences. Journal of American Society for Information Science, 32(January), 23–32.

Foulin, J. (1998). To what extent does pause location predict pause duration in adultsÕ and childrenÕs writing? Cahiers de Psychologie Cognitive, 17(3), 601–620.

Grunig, J. E. (1985). An axiomatic theoryof cognition and writing. Journal of Technical Writing and Communication, 15, 95–130.

Hebb, D. O. (1955). Drives and the conceptual nervous system. Physiology Behavior Review, 62, 243–254. Houston, J. P. (1981). Fundamentals of learning and memory. New York: Academic Press.

Kirby, M., & Miller, N. (1986). Medline searching on colleague: Reasons for failure or success of untrained end user. Medical Reference Services Quarterly, 5, 17–34.

Miller, G. A. (1956). The magic number seven, plus or minus two: Some limits of capacityfor processing information. Psychology Review, 63, 81–97.

Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliﬀs, NJ: Prentice-Hall. Norman, D. A. (1981). Categorization of action slips. Psychological Review, 88, 1–15.

Shaw, D. (1986). Nine sources of problems for novice online searchers. Online Review, 10, 295–303.

Tavalin, F. (1995). Context for creativity: Listening to voices, allowing a pause. Journal of Creative Behavior, 29(2), 133–142.

Tenopir, C. (1986). Four options for end user searching. Library Journal, 111, 56–57. Trzebiatowske, E. (1984). End user studyon BRS/after dark. RQ, 23, 446–450.

Yuan, W. (1997). End-user searching behavior in information retrieval: A longitudinal study. Journal of the American Society for Information Science, 48, 218–234.

Walker, C. J., McKibbon, K. A., Hayes, R. B., & Ramsden, M. F. (1991). Problems encountered by clinical and users of MEDLINE and GRATEFUL MED. Bulletin of Medical Library Association, 79, 67–69.

Wozny, L. A. (1988). College students as end user searchers: One universityÕs experience. RQ, 28, 54–61.