School of Information & Library Science, University of North Carolina at Chapel Hill 100 Manning Hall, CB #3360, Chapel Hill, NC 27599
[email protected]
ABSTRACT
Subjectivity is inherent in exploratory search. It is evident in the attributes of the searcher before beginning the search (e.g., the searcher’s domain knowledge). In the form of the searcher’s interests, knowledge, cognitive/learning style, and personality, it affects the search process – its overall trajectory. And it comes into play at the end of the search, when the searcher makes judgments about the success of the search. Researchers and designers interested in explora-tory search will need to develop measures of searchers’
beliefs about a search’s success and attitudes toward the search outcomes. These measures can then be leveraged to more directly evaluate the quality of the search process and to design personalized systems that improve a searcher’s performance.
INTRODUCTION
As White et al. [32] note, “defining what constitutes an exploratory search is challenging” [p.38]. Nevertheless, they provide an initial definition, in concert with that provided by Marchionini [21], that, “in exploratory search, users generally combine querying and browsing strategies to foster learning and investigation” [32, p.38]. In the “real world” (i.e., outside the boundaries of TREC-like evaluation paradigms), each exploratory search is conducted by an individual person or a small group of people. In this sense, the process of conducting an exploratory search is inherently subjective.
This inherent subjectivity can be addressed in many different ways. In some research traditions, the subjective experience is highlighted as the preferred focus of a particular social phenomenon. In these traditions, the researcher attempts to understand the phenomenon of interest from the participants’ perspectives, seeing it through their eyes. This approach rests on the assumption
that reality is, in fact, socially constructed and cannot be understood apart from the understandings of those partici-pating in its construction [5]. However, for others (include-ing most system designers), subjectivity introduces
“fuzziness” or complexity into our understanding of a particular reality [27, p.5], making it difficult to generalize study findings beyond the study participants.
The subjectivity inherent within a particular search trajec-tory has implications both for understanding the search process and for evaluating the effectiveness of a particular IR system in supporting the search process. Most impor-tantly, the inherent subjectivity of the search process may work against both the standardized evaluation of a system’s effectiveness and, possibly, even the design of systems that are useful to a large number of people. This paper will explore the implications of subjectivity at three points in an exploratory search: the starting point for the search, the trajectory of the search being undertaken, and the conclu-sion of the search. In addition, implications for evaluation methods and for system design will be discussed.
THE STARTING POINT FOR THE SEARCH
A variety of individual characteristics may affect a search at its beginning, but perhaps the most prominent of these is the domain knowledge of the searcher. The focus of exploratory search on learning and investigation makes it very likely that the searcher’s domain knowledge will influence retrieval success, as well as the search process.
Somewhat surprisingly, domain knowledge has not been consistently found to affect retrieval success. Studies finding a positive effect of domain knowledge on retrieval success include those by Jacobson and Fusani [19] and Marchionini [22]. However, a number of studies have found no such relationship [1, 4, 13, 26, 33]. Thus, whether domain knowledge affects the recall and precision of a retrieved set is still an open question.
A few studies have looked directly at the effect of domain knowledge on the process of searching, identifying the effects of domain knowledge on the tactics used in search strategy formulation and reformulation [34], the amount of time spent monitoring a search and the frequency with which terms were combined [18], the focus/attention of the searcher and his or her expectations for the outcomes of a
search terms [2, 18, 29, 30, 31]. Other studies have exam-ined the effects of domain knowledge on navigation in hypertext systems, and found that it affected use of links and topic abandonment [7], as well as search efficiency [24].
Wildemuth’s [34] study can be used as an example of an approach to understanding the effects of pre-existing domain knowledge on the search process. The searches conducted by the study participants were not defined as exploratory searches, but they might be considered as examples of this type, since students were using searches in a factual database in microbiology to help them solve simulated clinical problems (i.e., they were learning about the material in the database in preparation for their course exam and were simultaneously investigating the specific clinical problem described in the simulation). Figure 1 (columns labeled 1-3) illustrate the three tactics most often used for searches before the course, when the students’
domain knowledge was very low. The tactics shown in columns 1 and 2, as well as a tactic including only the selection of a new concept and a display of the results, were used at the end of the course (when domain knowledge was particularly high). Six months after the course ended, when domain knowledge had dropped somewhat, the study participants continued to use the tactic shown in column 1 and also used the tactic shown in column 4 (a slightly less efficient tactic than that developed in their searches just after the course).
Wildemuth’s [34] study, like most of the other studies cited above, measured domain knowledge before the search process was begun, and used that as a variable in later analyses of the search process or search results data. This approach is based on the assumption that domain knowledge is fairly stable and will not vary from the beginning to the end of the search episode. However, the definition of exploratory search implies that this assumption may be false. During exploratory search, it is expected that one of the outcomes of the search process is a change in the searcher’s knowledge of the domain brought about through encounters with new information during the course of the search process. Thus, domain knowledge can be expected to have even more pronounced effects on exploratory search than on look-up searches, and these effects will be manifest at the beginning of the search process and throughout the search process trajectory.
THE TRAJECTORY OF THE SEARCH
As noted above, the searcher’s domain knowledge as the search begins is likely to have an effect on the trajectory of a particular search. In addition, domain knowledge may be altered during a search. Consider an example. After attending a workshop on research ethics, a doctoral student saw an ABC “Primetime Live” show and became interested in Milgram’s experiments in the 1960’s. She picked up the transcript of the show from Lexis Nexis, but decided to look further. A quick google search sent her to Wikipedia, where she learned a bit more background and followed several links to other Wikipedia articles. The google search also sent her to a number of other sites about Milgram and his experiments on obedience. She then went back to Psychological Abstracts and found a review of a book by Arthur Miller that discusses the Milgram experiments as a case study in methodological controversy. This search concluded with her going to the library and checking out Miller’s book. A more fine-grained look at this example of an exploratory search would include her interactions with (i.e., the tactics used in) each formal information source (Lexis Nexis, google, Psychological Abstracts, the library catalog). The searcher’s knowledge about the Milgram experiments was meager at the beginning of the process, but by the time she saw the review of Miller’s book, she had a good understanding of the original experiments and people’s objections to them. Her level of domain knowledge evolved during the search process, enabling her to make “stopping” decisions in google and Psychological Abstracts when she knew enough to move in a different direction. The search trajectory was likely affected by the student’s interests in developing her own research career in the social sciences, her own pre-existing domain knowledge, her ability to learn during a search, and aspects of her personality (e.g., the curiosity and persistence found in many doctoral students). If this story had a different main character, it is likely that it would have a different plot. This difference reflects the influence of subjectivity on the trajectory of a search.
Search trajectories have been investigated to some extent.
Several of the studies cited above include explicit analysis of the steps taken during a search. Additional studies have examined search tactics more generally. For example, Rieh and Xie [25] identified eight distinct patterns of search modification sequences when they examined 313 Web searches. Clark et al. [9] integrated the hypertext navi-gation patterns identified by Canter et al. [6] – path, ring, loop, and spike – with the general browsing strategies that people use (e.g., scanning would involve a mix of deep spikes and short loops). Curzon, Wilson, and Whitney [10]
examined older adults’ use of both print-based information resources and the Web, to see the ways in which each type of source was used. They found that this group had well-rehearsed tactics for use in traditional sources which did not always transfer gracefully to the Web.
In addition to general studies of search processes, two re-cent studies have investigated the effects of individual dif-ferences on the search trajectory. Ford, Miller and Moss [15] found that several aspects of cognitive style and the searcher’s ability to deal with cognitive complexity affected use of a Boolean search strategy versus a best-match search strategy; they also found that age and gender were related to choice of search strategy. Graff [17] also found that cognitive style and age were related to the number of pages visited in two different hypertext architectures, though they were not related to the browsing strategy (surface versus deep) used. Based on these results, additional studies are needed to identify which individual characteristics have the strongest influence on the search process.
THE CONCLUSION OF THE SEARCH
As a search is concluded, the searcher will make judgments about whether it has been successful. For exploratory searches, one would expect that those judgments would be based on whether the searcher learned enough to be satisfied and/or whether the question being investigated was satisfactorily resolved. In other words, the searcher’s judgment about search success will be based on the information used in accomplishing his or her goals, not just on the information retrieved during the course of the search.
For researchers, the challenge posed by these judgments is that they are based on goals that are likely to be changing throughout the course of the search – particularly for exploratory searches. In the story in the previous section, the student began with the goal of learning more about the Milgram experiments. By the time she started reading Miller’s book, her goal was to learn more about how meth-odological controversies in the social sciences are resolved.
If one takes an extreme position, one would say that a search can not be considered successful unless the search goals have changed as a result of learning that occurred during the search process. Thus, it is not clear which goal should be used in an evaluation, or whether some type of
“goal-free” evaluation [28] should be conducted.
IMPLICATIONS FOR EVALUATION AND DESIGN
From the discussion above, it is clear that traditional measures of retrieval success (e.g., precision and recall) will not be appropriate for evaluating the success of exploratory searches. Yet some type of reliable and valid measurements must be taken if we are to compare the effectiveness of one system in supporting exploratory searches with the effectiveness of another system.
A first step may be to rely on users’ perceptions of success as a basis for evaluation. There are at least two possibilities here: to ask users for their judgments of the success of a search or to ask users about their satisfaction with the outcomes of the search. While both of these measures might rely on some type of questionnaire, each has a different focus [12]. The first is asking people about their beliefs, or their assessments of what they think is true or false.
Specifically, it would ask people for their judgments about whether the search was successful or not, and is simply a subjective measure of system performance. The second type of questionnaire is asking people about their attitudes, or how they feel about the search outcomes. This type of measure has an affective component not present in a questionnaire focused on beliefs.
At present, the field has no reliable and valid questionnaire that can be used for either of these measures;1 development of such questionnaires would be a significant contribution to the field. In the meantime, some measures of aspects of user beliefs or attitudes might be applied, including Davis’
[11] measure of perceived usefulness, Doll and Torkzadeh’s [14] measure of satisfaction, or other measures adapted from their use in the fields of information systems or human-computer interaction.
If we can find a way to evaluate search outcomes, then we could leverage those measures to better understand which search processes are most effective. For example, in a given study, the search tactics used by the most successful searches could be analyzed and compared with the tactics used in less successful searches. Based on these types of comparisons, we may be able to identify particular patterns that lead to better outcomes, thus enabling direct evaluation of search processes. In addition, we could evaluate users’
satisfaction with the search process, including measures of satisfaction with the user-system interaction [8] or measures of flow [16] and other affective responses.
Being able to evaluate the success of a search also provides leverage for improving retrieval system design. Given the subjective character of exploratory search, it seems likely that personalization will play a key role in the design of systems intended to support such searches. The user
1 An alternative – an auction based on the perceived value of a system – has been proposed by Ben-Bassat, Meyer, and Tractinsky [2006]. They argue that bids placed in such an auction are highly related to user performance.
attributes that will be the most effective levers of personalization will be discovered most readily only when we have a means for telling a successful search from an unsuccessful search. Thus, developing valid evaluation methods is critical to progress in the field.
REFERENCES
1. Allen, B. Topic knowledge and online catalog search formulation. Library Quarterly 61, 2(1991), 188-213.
2. Allen, B. Logical reasoning and retrieval performance.
Library & Information Science Research 15 1(1993), 93-105.
3. Ben-Bassat, T., Meyer, J., and Tractinsky, N. Economic and subjective measures of the perceived value of aesthetics and usability. ACM Transactions on Computer-Human Interaction 13, 2(2006), 210-234.
4. Borgman, C.L., et al. Children’s searching behavior on browsing and keyword online catalogs: The Science Library Catalog Project. Journal of the American Society for Information Science, 46, 9(1995), 663-684.
5. Bradley, J. Methodological issues and practices in qualitative research. Library Quarterly 63, 10(1993), 431-449.
6. Canter, D., Rivers, R. and Storrs, G. Characterizing user navigation through complex data structures. Behaviour &
Information Technology 4, 2(1985), 93-102.
7. Carmel, E., Crawford, S., and Chen, H. Browsing in hypertext:
A cognitive study. IEEE Transactions on Systems, Man, and Cybernetics 22, 5(1992), 865-884.
8. Chin, J.P., Diehl, V.A., and Norman, K.L. Development of an instrument measuring user satisfaction of the human-computer interface. CHI ’88 Proceedings, (1988), 213-218.
9. Clark, L., Ting, I., Kimble, C., Wright, P. and Kudenko, D.
Combining ethnographic and clickstream data to identify user Web browsing strategies. Information Research 11, 2(2006), paper 249. Available at http://InformationR.net/ir/11-2/paper249.html.
10. Curzon, P., Wilson, J., and Whitney, G. Successful strategies of older people for finding information. Interacting with Computers 17, 6(2005), 660-671.
11. Davis, F.D. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13, 3(1989), 319-340.
12. Dillman, D.A. Mail and Telephone Surveys: The Total Design Method. John Wiley & Sons, New York, NY, 1978.
13. Dimitroff, A., and Wolfram, D. Searcher response in a hypertext-based bibliographic information retrieval system.
Journal of the American Society for Information Science 46, 1(1995), 22-29.
14. Doll, W.J., & Torkzadeh, G. The measurement of end-user computing satisfaction. MIS Quarterly 12, 2(1988), 259-276.
15. Ford, N., Miller, D., and Moss, N. Web search strategies and human individual differences: Cognitive and demographic factors, internet attitudes, and approaches. Journal of the American Society for Information Science & Technology 56, 7(2005), 741-756.
16. Ghani, J.A., Supnick, R., and Rooney, P. The experience of flow in computer-mediated and in face-to-face groups. Proc., 12th Int. Conf. on Information Systems, (1988), 229-237.
17. Graff, M. Individual differences in hypertext browsing strate-gies. Behaviour & Information Technology 24, (2005), 93-99.
18. Hsieh-Yee, I. Effects of search experience and subject knowledge on the search tactics of novice and experienced searchers. Journal of the American Society for Information Science, 44, 3(1993), 161-174.
19. Jacobson, T., and Fusani, D. Computer, system, and subject knowledge in novice searching of a full-text multifile database. Library & Information Science Research 14, 1(1992), 97-106.
20. Marchionini, G. Exploratory search: From finding to under-standing. Communications of the ACM 49, 4(2006), 41-46.
21. Marchionini, G. Information-seeking strategies of novices using a full-text electronic encyclopedia. Journal of the American Society for Information Science 40, 1(1989), 54-66.
22. Marchionini, G., Dwiggins, S., Katz, A., and Lin, X.
Information seeking in full-text end-user-oriented search systems: The roles of domain and search expertise. Library &
Information Science Research 15, 1(1993), 35-69.
23. McDonald, S., and Stevenson, R.J. Navigation in hyperspace:
An evaluation of the effects of navigational tools and subject matter expertise on browsing and information retrieval in hypertext. Interacting with Computers 10, 2(1998), 129-142.
24. Rieh, S.Y., and Xie, H.(I.) Analysis of multiple query reformulations on the Web: The interactive information retrieval context. Information Processing & Management 42, (2006), 751-768.
25. Saracevic, T., and Kantor, P. A study of information seeking and retrieving. III. Searchers, searches, and overlap. Journal of the American Society for Information Science 39, 3(1988), 197-216.
26. Schwartz, H., and Jacobs, J. Qualitative Sociology: A Method to the Madness. Free Press, New York, NY, 1979.
27. Scriven, M. Goal-free evaluation. In House, E.R. (ed.), School Evaluation: The Politics and Process. McCutchan, Berkeley, CA, 1973.
28. Shute, S.J., and Smith, P.J. Knowledge-based search tactics.
Information Processing & Management 29, 1(1993), 29-45.
29. Vakkari, P. Subject knowledge, source of terms, and term selection in query expansion: An analytic study. Advances in Information Retrieval: 24th BCS-IRSG European Colloquium on IR Resarch, Proceedings (2002), 110-123.
30. Vakkari, P., Pennanen, M., and Serola, S. Changes of search terms and tactics while writing a research proposal: A longitudinal case study. Information Processing &
Management 39, 3(2003), 445-463.
31. White, R.W., Kules, B., Drucker, S.M., and schraefel, m.c.
Supporting exploratory search. Communications of the ACM 49, 4(2006), 37-39.
32. Wildemuth, B.M., de Blied, R., Friedman, C.P., and File, D.D.
Medical students’ personal knowledge, searching proficiency, and database use in problem solving. Journal of the American Society for Information Science, 46 (1995), 590-607.
33. Wildemuth, B.M. The effects of domain knowledge on search tactic formulation. Journal of the American Society for Information Science & Technology 55, 3(2004), 246-258.