Collaborative relevance assessment for task-based knowledge support

(1)

Collaborative relevance assessment for

task-based knowledge support

Duen-Ren Liu

a,

⁎

, I-Chin Wu

b a

Institute of Information Management, National Chiao Tung University, Hsinchu 300, Taiwan b_{Department of Information Management, Fu Jen Catholic University, Taipei 242, Taiwan}

Received 16 April 2004; received in revised form 20 April 2007; accepted 20 June 2007 Available online 27 June 2007

Abstract

The operations and management activities of enterprises are mainly task-based and knowledge intensive. Accordingly, an important issue in deploying knowledge management systems is the provision of task-relevant information (codified knowledge) to meet the information needs of knowledge workers during the execution of a task. Codified knowledge extracted from previously executed tasks can provide valuable knowledge about conducting the task-at-hand (current task), and is a valuable information source for constructing a task profile that models a worker's task needs, i.e., information needs for the current task. In this paper, we propose a novel task-relevance assessment approach that evaluates the relevance of previous tasks in order to construct a task profile for the current task. The approach helps knowledge workers assess the relevance of previous tasks through linguistic evaluation and the collaboration of knowledge workers. In addition, applying relevance assessment to a large number of tasks may create an excessive burden for workers. Thus, we propose a novel two-phase relevance assessment method to help workers conduct relevance assessment effectively. Furthermore, a modified relevance feedback technique, which is integrated with the task-relevance assessment method, is employed to derive the task profile for the task-at-hand. Consequently, task-based knowledge support can be enabled to provide knowledge workers with task-relevant information based on task profiles. Empirical experiments demonstrate that the proposed approach models workers' task-needs effectively and helps provide task-relevant knowledge. © 2007 Elsevier B.V. All rights reserved.

Keywords: Information retrieval; Knowledge management; Relevance assessment; Task-based knowledge support; Task profile

1. Introduction

In organizations, knowledge management (KM) is an important means of gaining a competitive advantage. To this end, knowledge management systems (KMS) maximize the effectiveness of knowledge assets, thereby increasing an organization's profitability and productiv-ity [17,27]. KMS employs information technologies (IT), such as document management and data mining, to

facilitate access to, sharing of, and reuse of knowledge

assets within and across organizations[8,22].

Intellectual content is generally codified in an explicit

form to facilitate knowledge sharing and reuse[5,26,38].

Textual data, such as articles, reports, and manuals, is treated as valuable and explicit knowledge in organiza-tions. Codifying structured and explicit knowledge into a knowledge repository, especially in document form, is a commonly used strategy for managing knowledge

[8,16,38]. Empirical findings indicate that codifying intellectual content into a knowledge repository helps workers exploit existing organizational resources Decision Support Systems 44 (2008) 524–543

www.elsevier.com/locate/dss

⁎ Corresponding author. Tel.: +886 3 5131245; fax: +886 3 5723792. E-mail address:[email protected](D.-R. Liu).

(2)

efficiently [15]. Accordingly, knowledge (information) retrieval is a core component of KMS to retrieve codified knowledge. An effective knowledge retrieval function can mitigate the difficulty of obtaining knowledge items from

a knowledge repository[11,14]. Information retrieval (IR)

techniques are widely used to implement knowledge retrieval functions. Query-based information retrieval is a user-driven approach that accesses knowledge items by translating user information needs into compromised queries. Alternatively, information filtering based on profile construction provides a system-driven approach

for proactive delivery of relevant information to users[4].

Since a company's operational and management activities are mainly task-based, workers perform various tasks to achieve business goals. In task-based business environments, an important issue of deploying KMS is the provision of task-relevant information (codified knowl-edge) to meet the information needs of knowledge workers during the execution of a task. In recent years, information retrieval techniques coupled with workflow management systems (WfMS) have been used to support proactive delivery of task-specific knowledge according

to the context of tasks within a process [1,2,11,12]. For

example, the KnowMore system maintains task specifica-tions (profiles) that define the process-context of tasks and

associated knowledge items [1]. Thus, context-aware

delivery of task-specific knowledge can be provided based on the task specifications and the execution context of the current process. The Kabiria system supports knowledge-based document retrieval in office environments by allowing users to retrieve documents according to the

operational context of task-associated procedures[6]. In

[23], a process meta-model that specifies the knowledge in

context is integrated with workflow systems to capture and retrieve knowledge within a process context.

The above works provide an appropriate perspective for designing task-based knowledge support. However, they focus on specifying the process-context of a task to support context-aware or process-aware knowledge re-trieval, rather than on a systematic method for constructing a task profile that models a worker's task needs, i.e., information needs for the current task (task-at-hand).

In this work, we focus on providing codified knowledge support for knowledge-intensive tasks within organizations. Examples of knowledge-intensive tasks include thesis-writing and research projects in academic organizations, project management in firms, and research and product development in R&D departments. Because of the nature of knowledge-intensive tasks, a collaborative mechanism is important for developing a knowledge

support system [3,40]. Moreover, huge amounts of

codified knowledge place an excessive burden on

knowledge workers. Intelligent search engines, agent-based techniques, and information filtering have been applied to deliver information relevant to a worker's task-at-hand. Knowledge sharing and intelligent search services support knowledge management in a

decentra-lized global business[34]. Agents cooperate to achieve

task-based information filtering within a work process based on user feedback about a document's usefulness to

a particular job situation[9]. Alternatively, agents can use

inference engines to reason proactively about the needed

information in order to provide effective task support[35].

Information filtering with a similarity-based approach is often used to locate knowledge items relevant to the task-at-hand, whereby the relevance of a knowledge item is determined by analyzing the similarity between the concept terms of the knowledge item and the current task

[19,37]. The concept terms are usually extracted from the textual descriptions of a knowledge item/task. For example, if the task-at-hand is the development of a software program, the task needs to be a partially written

program with comments[37]. However, the major issue

is that the comments may not describe the task precisely and completely, so the similarity-based approach may not be able to locate relevant information (software

compo-nents). Holz et al.[19]propose a task-oriented,

similarity-based approach that organizes desktop documents and proactively delivers task-specific information. The effec-tiveness of this approach also relies on the accuracy of the concept terms of the task-at-hand, which are extracted from the task name and associated relevant documents.

Even though the above works support task execution by using information filtering techniques, they cannot effec-tively model a worker's initial task-needs (i.e., information needs for the initial phase of a task) when very few or no textual task-descriptions exist for extracting the concept terms of the task-at-hand. Similarity analysis based on concept terms cannot derive appropriate information to meet initial task needs, since the concept terms of the task-at-hand extracted from the textual descriptions do not properly represent the task. In this work, we attempt to resolve the problem by employing a user-based assessment approach to model a worker's initial task-needs.

Historical codified knowledge items, task descriptions, and relevant documents extracted from previously executed tasks provide valuable knowledge sources for supporting task profile construction. Rather than ask knowledge workers to specify task characteristics directly, a systematic approach is preferable to create a task profile, i.e., the concept terms of the current task (task-at-hand), based on the concept terms and the relevance of previous tasks. However, for tasks with very few or no textual descriptions, user assessment is necessary to determine

(3)

the relevance (similarity) of previous tasks to the current task. Accordingly, we propose a novel task-relevance assessment approach to evaluate a task's relevance in order to construct task profiles that model the worker's information needs for the current task. A modified relevance feedback (RF) technique is employed to derive the task profile based on the degree of relevance and the concept terms of previous tasks. The approach helps knowledge workers assess the relevance of previous tasks through linguistic evaluation and the collaboration of knowledge workers. Even so, applying relevance assess-ment to a large number of tasks may place an excessive burden on workers. Therefore, we propose a novel two-phase relevance assessment method to help workers conduct relevance assessment effectively. Consequently, task-based knowledge support can be enabled to provide knowledge workers with task-relevant information based on task profiles. Our empirical experiments demonstrate that the proposed approach is effective in providing task-based knowledge support.

The remainder of the paper is organized as follows. Section 2 presents an overview of our approach and describes the architecture of the proposed task-based knowledge support system. Section 3 introduces the basic techniques used in this work. Section 4 describes the process of building the task-oriented repository. Section 5 describes the two-phase collaborative rele-vance-assessment procedure for generating task profiles. The experiment evaluations are reported in Section 6. Finally, in Section 7, we present our conclusions and indicate the direction of our future work.

2. Task-based knowledge support

In this section we present an overview of the proposed approach, and then describe the architecture of the proposed task-based knowledge support system. 2.1. Overview of task-based knowledge support

Knowledge workers generally require historical task-relevant information to accomplish current tasks. Thus, reusing knowledge about previous tasks is the key to providing effective knowledge support for conducting new tasks. Accordingly, the proposed approach manages an organization's codified knowledge by using a task-based categorization scheme to organize tasks into categories.

Codified knowledge is analyzed, categorized, and stored in a knowledge repository. This work broadly defines a task as a unit of work, such as a research project in an organization. A task denotes either an executed task or a current task. An executed-task is a historical

task already accomplished within the organization, whereas a current-task is the task at hand. Categories representing the main subjects of organizational activi-ties are defined in order to organize tasks and codify knowledge. A task profile specifies the key concept terms of the current task, and models the information needs of knowledge workers during the task's execution. A task corpus specifies the key concept terms of an executed task. Reference tasks are a subset of executed tasks selected for the construction of a task profile.

As noted earlier, large amounts of codified knowledge place an excessive burden on knowledge workers. Previous works have attempted to solve the problem with information filtering techniques. However, they cannot effectively model a worker's initial task-needs, when very few or no textual descriptions exist to properly represent the current task. In contrast, our approach constructs a task profile that models a worker's information needs for the current task based on the corpus (concept terms) and the relevance of previously executed tasks. A novel task-relevance assessment approach is proposed to determine the relevance of previously executed tasks to the current-task. We then use a modified relevance feedback (RF) technique to derive the task profile based on the degree of relevance and the concept terms of executed tasks.

2.1.1. Task relevance assessment

The proposed approach addresses the following three issues that arise during user assessment.

First, assessing the relevance of a task by assigning precise numerical values may be difficult for knowledge workers. Thus, a fuzzy linguistic approach, an

approx-imate technique that models human thinking[39], is used

to evaluate a task's relevance by using linguistic terms

such as “low” or “high” to express the worker's

perception of“Relevance”. The proposed task-relevance

assessment method provides a systematic and natural way to analyze the relevance of tasks in the repository. The fuzzy linguistic approach is described in Section 3.3. Second, for complex and knowledge-intensive tasks, collaboration among knowledge workers and experts is often necessary to facilitate more effective knowl-edge dissemination. In addition, novices who are less knowledgeable about the current task may have difficulty in assessing the relevance of tasks. Domain experts or experienced workers with valuable implicit knowledge play an important role in helping knowledge workers

solve problems or make decisions[12–14]. Accordingly,

the proposed assessment approach incorporates a collab-orative mechanism that helps knowledge workers, especially novices, determine the relevance of executed

(4)

tasks. Herein, collaboration means that knowledge workers (e.g., experts and task members) can exchange the evaluation results of knowledge items (e.g., topic taxonomy, task sets, etc.) via a common interface (i.e., task assessment editor) in the proposed task-based workspace. This is similar to the concept of collaborative information retrieval proposed by Hansen and Jarvelin

[18] in that we focus on information access behavior

related to a specific problem-solving activity, which requires interaction among workers in a common workplace. A collaborative mechanism, on the other hand, refers to the process of conducting assessments and aggregating the relevance results of workers and domain experts to derive a task's relevance in a collaborative workplace. The concept is similar to the collaboration

module in the COPLINK workflow model[40]. In this

work, however, we activate the collaborative mechanism by loading the evaluation results at the start of the task's execution in order to model a worker's initial task-needs for constructing task profiles, rather than identifying similar search cases to support the search results based on the user's search actions.

Third, applying relevance assessment to a large number of previously executed tasks may create a burden for workers and influence the assessment result. Thus, a novel two-phase relevance assessment procedure is used to reduce the number of tasks to be assessed by conducting category assessment to select a subset of executed tasks as reference tasks for further task assessment. A task categorization scheme is used to group tasks into categories. Identifying a small subset of executed tasks as reference tasks can help knowledge workers conduct further task-relevance assessment without reviewing all previously executed tasks. The reference tasks are selected based on their similarity to the current task using the degree of relevance of a task to the categories.

2.1.2. Generating task profiles by modified relevance feedback techniques

The reference tasks are used to extract task-relevant knowledge for the current task. Once the reference tasks have been identified, a modified relevance feedback (RF) technique is used to derive the task profile of the current task based on the degree of relevance and the concept terms of the reference tasks. The knowledge support system uses task profiles to identify relevant information and assist knowledge workers in accessing task-relevant knowledge during the execution of the current task. Relevance feedback, a well-known tech-nique in information retrieval, improves the search effectiveness by automatically reformulating queries

[30]. The relevance feedback techniques are described

in Section 3.2. Most traditional RF techniques use the concept terms of relevant/irrelevant documents to adjust or reformulate the concept terms of a query. The concept terms of relevant documents have a positive influence on the reformulation, while those of irrelevant documents have a negative influence. The relevance of documents is based on user feedback about the previous query result. We adopt the RF technique to derive the task profiles. However, unlike traditional RF techniques that use direct user feedback about previous query results, the proposed two-phase relevance assessment procedure helps work-ers determine a task's relevance based on the proposed categorization scheme. In addition, our modified RF approach uses aggregated relevance ratings as the degrees of relevance of reference tasks to adjust the task profile, whereas most RF techniques consider binary relevance, i.e., relevant/irrelevant.

2.2. System architecture of task-based knowledge support

Fig. 1illustrates the architecture of the proposed task-based knowledge support system. The participants are knowledge workers engaged in specific tasks and domain experts in specific subjects. The system comprises three main modules: the task-oriented infor-mation repository, the task profile handler, and the task-oriented retrieval router.

Task-oriented information repository. This module is the knowledge base for task-based knowledge support. Task-oriented repositories are constructed with support from the task-based categorization scheme to ensure that codified knowledge is utilized effectively. The reposito-ries, which store codified knowledge corresponding to a task's execution, contain three databases: the document-indexing database, the task corpus, and the task categorization database. The document-indexing data-base stores task relevant documents indexed by their concept terms. The key concept terms of knowledge items (documents) are represented as a feature vector of weighted terms using the IR techniques described in Section 3.1. A task corpus contains the key concept terms of an executed task, and is expressed as a feature vector of weighted terms. The task corpus of an executed task is generated by extracting the weighted terms from textual documents of the executed task. This process is described in detail in Section 4.1. Finally, the task categorization database records the relationships be-tween previously executed tasks and categories; that is, the relevance of executed tasks to the categories. The task categorization database helps identify reference tasks based on their similarity to the current task.

(5)

Task profile handler. The task profile handler comprises the following mechanisms for profile man-agement: profile creation, profile adjustment, and profile adaptation. A task-profile specifies the key concept terms of the current task and is expressed as a feature vector of weighted terms. Task-profiles form the basis for discovering and disseminating task-relevant information to knowledge workers. The system generates profiles based on the task corpus of relevant executed tasks and employs the proposed assessment mechanism for task profile adjustment.

Task-oriented retrieval router. The task-oriented retrieval router matches task profiles with knowledge repositories to streamline knowledge retrieval. The router retrieves and disseminates task-relevant informa-tion to provide task-based knowledge support based on task profiles.

2.2.1. Process of task-based knowledge support

Fig. 1 also shows the process of the relevance assessment and knowledge support. The task plan or textual descriptions of the current task can be used to extract the concept terms of the task to construct an initial task profile. However, the profile may not effectively model the initial task needs, especially for tasks with very few or no textual descriptions. Using a fuzzy linguistic approach, a two-phase assessment process evaluates a task's relevance based on the collaboration of knowledge workers and domain experts. Knowledge workers can assess the relevance of previously executed

tasks to the current task through the collaborative task-based workplace of our system. The system's assessment interface supports linguistic relevance ratings. In addi-tion, the workplace shows the relevance ratings given by other collaborative workers (experts or task-relevant colleagues). The worker may adopt a collaborative assessment, where the relevance ratings of other workers are combined with his/her own relevance ratings to obtain aggregated relevance ratings.

The task categorization database contains the categorization scheme, including the categories and the degree of relevance of executed tasks to the categories, to support the proposed two-phase task-assessment pro-cess. The worker conducts category assessment (phase-1 assessment) based on the category scheme to derive the relevance of the current task to categories. The system then selects a set of reference tasks from previously executed tasks based on the similarity measures derived according to the degree of relevance of executed/current tasks to categories. The reference tasks are displayed on the assessment interface to help workers conduct further task assessments. Thus, the knowledge worker can conduct a task assessment (phase-2 assessment) without reviewing all previously executed tasks. The two-phase relevance assessment procedure is discussed in detail in Section 5.1.

The assessment results and the initial profile are used to generate the worker's task profile based on the modified relevance feedback technique. The technique considers the aggregated relevance ratings and the task Fig. 1. The architecture of the task-based knowledge support system.

(6)

corpus of reference tasks to refine the initial task profile. Details are given in Section 5.2. The generated task profile is stored in the task profile database located in the information repository for future knowledge retrieval.

When a worker submits a query to retrieve task-relevant item sets, the task-oriented retrieval router retrieves the task profile of current task from the information repository based on the task-query. The router then matches the task profile with codified knowledge items stored in the repository to retrieve task-relevant items (i.e., relevant documents and tasks) according to the similarity measures of the task profile and the knowledge items. It then disseminates the task-relevant items to workers to provide task-based knowl-edge support. Note that all text-related processing, including information extraction and text pre-processing, is performed off-line in the system's back-end by the text-processing server module.

Section 4 describes the construction of a task-oriented information repository. Textual data is analyzed and stored in the task-oriented information repository. Section 5 describes the proposed relevance-assessment approach for generating and adjusting task profiles. 3. Preliminary techniques

This section briefly reviews some basic techniques, including information retrieval, the relevance feedback technique, and the fuzzy linguistic approach. In this work, we modify and combine these methods to support task-relevance assessment and construct task profiles. Since retrieving knowledge items from textual data is our primary objective, we adopt information retrieval and information filtering techniques for text pre-processing, indexing, querying, and profiling tasks. Furthermore, using a fuzzy linguistic approach, we incorporate linguistic ratings into the relevance feedback technique. 3.1. Information retrieval in a vector space model

The key contents of a codified knowledge item (document) can be represented as a feature vector of weighted terms in n-dimensional space, using a term weighting approach that considers term frequency, inverse document frequency, and normalization factors

[31]. The term transformation steps, i.e., case folding,

stemming, and stop word removal, are performed during

text pre-processing[29,33,36]. Then, term weighting is

employed to extract the most discriminating terms[4].

Let d be a codified knowledge item (document), and

Y_d _{¼ hwðk}

1; dÞ; wðk2; dÞ; N ; wðkn; dÞi be the feature

vector of d, where w(ki, d) is the weight of a term ki

that occurs in d. The weight of a term indicates its degree of importance in representing the document (codified knowledge). The well-known tf-idf approach,

which is often used for term (keyword) weighting[29],

assumes that terms that occur more frequently in one document compared to other documents are better discriminators for representing that document. Let the

term frequency tf(ki,d) be the frequency that term ki

occurs in d, and let the document frequency df (ki)

represent the number of documents that contain term ki.

The importance of term kito a document d is proportional

to the term frequency and inversely proportional to the document frequency, as expressed in Eq. (1).

wðki; dÞ ¼ 1 ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X i ðtf ðki; dÞ logðN=df ðkiÞÞÞ2 r tfðki; dÞ log N dfðkiÞ ; _ð1Þ

where N is the total the number of documents . The denominator on the right-hand side of the equation normalizes the weight of a term.

3.1.1. Similarity measure

The cosine formula is a similarity measure that is widely used to assess the degree of similarity between two items, x and y, by computing the cosine of the angle

between their corresponding feature vectors, Yx and Yy, as

shown in Eq. (2). The degree of similarity is higher if the cosine similarity is close to 1.

simðx; yÞ ¼ cosineðYx; Yy Þ ¼ Yx

Yy

jYxj jYyj ð2Þ

Each document or query/task can be represented as

feature vector in a vector space model. Let Ydjrepresent a

vector of a document djand let Yq be a vector of a query/

task q. The similarity between a document dj and a

query/task q, sim(dj, q), can be calculated by Eq. (2).

3.2. Relevance feedback techniques

Relevance feedback (RF) improves the search

effec-tiveness through query reformulation [32]. The RF

technique reformulates or expands the original query based on partial relevance judgments, i.e., feedback on part of the evaluation set. Relevant documents with positive feedback have a positive influence on the weight of terms, while irrelevant documents with negative feedback have a negative influence on the weight of terms. A refined query vector can be generated by adding

(7)

the term weights of relevant documents and subtracting the term weights of irrelevant documents. Eqs. (3) and (4)

illustrate two classic relevance feedback methods— the

standard_Rocchio and the Ide_Dec_Hi methods designed

by Rocchio[30] and Ide[20], respectively. A modified

query vector Yq_m is derived using the relevance of

documents (as feedback) to adjust the query vector Yq[4].

Standard_{Rocchio : Y}q_m¼ aYqþ b 1 jDrj X 8dja Dr Y_d j g 1 jDnj X 8dja Dn Y_d j ð3Þ Ide_{DecHi : Y}q_m¼ aYqþ b X 8dja Dr Y_d j

g maxirrelevantðYdjÞ ð4Þ

where Drdenotes the set of relevant documents and Dn

represents the set of irrelevant documents according to

user assessments. |Dr| and |Dn| represent the number of

documents in the sets Drand Dnrespectively; andα, β, γ

are tuning constants. The function maxirrelevantreturns the

most irrelevant document. The two methods produce similar results[4].

We modify the standard_Rocchio and Ide_Dec_Hi methods by integrating them with the fuzzy linguistic approach to derive the aggregated degrees of relevance obtained from user assessments (discussed in Section 5.1). The modification considers the relative importance of relevant and irrelevant codified-knowledge from the user's perspective. The details are presented in Section 5.2. 3.3. Modeling user perceptions by a fuzzy linguistic approach

Generally, assessing qualitative problems by assign-ing precise values to them is difficult. A fuzzy lassign-inguistic approach, which approximates human perception, makes

it easier to evaluate qualitative problems[39]. Linguistic

assessment is based on words rather than numbers. A linguistic variable with linguistic terms and their associated meanings needs to be defined for linguistic assessment. A formal definition of a linguistic variable

can be found in[39].

In this work, a linguistic variable, Relevance, is defined to represent the degree of relevance between items (tasks or categories) assessed by workers. The

linguistic terms “very low”, “low”, “normal”, “high”,

“very high”, and “perfect”, are used to express the

context of“Relevance”. Let E(Relevance) represent the

linguistic terms of the linguistic variable Relevance. E

(Relevance) is characterized using a fuzzy set of the universe of discourse U = [0,1], in which six linguistic

terms,řj , and their associated semantic meanings, m(řj),

are defined as follows: E(Relevance) = {ř0=Very Low

(VL), ř1=Low (L), ř2=Normal (N), ř3=High (H),

ř4 = Very High (VH), ř5 = Perfect (P)}, where m(ři) b m(řj), for i b j, and all m(řj) are distributed in the range [0,1].

The fuzzy linguistic approach models the semantic

meaning of each term by fuzzy numbers[10]. This work

employs triangular fuzzy number (TFN), as defined in Appendix A, to express the approximate value of each linguistic term. A triangular fuzzy number is specified by

three parameters (l, m, r), with lbmbr, which determine

the x-coordinates of the three corners of the triangular membership function.

4. Task-oriented information repository

This section describes the two phases of constructing a task-oriented information repository: extracting the task corpus from textual data gathered during a task's execution, and determining the degrees of relevance between executed tasks and categories.

4.1. Extracting the task corpus

The task corpus of an executed task tris represented

as a feature vector of weighted terms derived by analyzing the set of documents generated and accessed

by tr. Each document djis pre-processed and represented

as a feature vector dYj, as described in Section 3.1. A

centroid approach is used to derive the feature vector of a task by averaging the feature vectors of documents

generated/accessed by the task. Let Dtrdenote the set of

documents generated/accessed by task tr. The task

corpus (feature vector) of task tr is defined as the

centroid vectorYtr, which is obtained by averaging the

feature vectors of documents in Dtr. Eq. (5) defines the

centroid vector Ytr. The weight of a term ki in Ytr is

represented by w(ki, tr). Y_t_r_¼ 1 jDtrj X dja Dtr Y_d j ð5Þ

4.2. Task categorization model

Previously executed tasks are categorized such that they may belong to more than one category. The task categorization database records the relationships be-tween executed tasks and categories, namely, the degree of relevance of an executed task to a category. The

(8)

degree of relevance indicates how well the task fits that category, and is calculated according to the similarity measures between the feature vectors of the categories and the executed tasks. The feature vector of a category is also expressed as a vector of weighted terms, which represents the main subjects of a category.

The categorization procedure comprises two steps: 1) deriving the feature vectors of categories; and 2) deriving the degrees of relevance between executed tasks and categories.

4.2.1. Deriving the feature vector of each category Experts predefine a set of categories to represent the main subjects in the organizational domain. A seed-based approach is then applied to generate the feature vectors of categories. Experts select some previously executed tasks as seed tasks to represent a category. A centroid vector can be derived from the corpus (feature vectors) of seed tasks to denote the category by averaging the feature vectors of corresponding seed tasks.

Let X denote a set of categories, X = {c1, c2,…, cm}, and let Tcjrepresent the set of seed tasks of category cj. Also let

Y

cc

j be the centroid vector derived from the task corpus

(feature vectors) of seed tasks in cj. The centroid weight of

the term kiincYcj, w(ki,Yccj) is derived by Eq. (6).

wðki;cYcjÞ ¼ 1 jTcjj X tra Tcj wðki; trÞ ð6Þ

The centroid vectors are used as the initial feature vectors of weighted terms to represent categories. The initial centroid weight of a term represents the degree of importance of the term in a category, without considering its importance in other categories; that is, its power to discriminate between categories. The weight of a term is further adjusted by considering its discriminating power. Common terms may not be discriminating enough to represent each category, even though they have high weights in some categories. To reduce the weight of such terms, we use the probability distribution of terms across categories to discriminate between the categories. To do this, we adjust the weight of a term in a category by multiplying its initial centroid weight by the probability distribution of the term appearing in the category.

Let Ycj be the feature vector of category cj , which

denotes the key concepts of cj, and let w (ki, cj) be the

weight of term ki in category cj. Then w (ki, cj), the

importance of term ki in representing category cj, is

proportional to the centroid weight of term ki and the

probability distribution of term kiappearing in category

cj, which is expressed as Eq. (7). Note that P(ki, cj) is the

probability distribution of term kiappearing in category

cj , which is computed according to the distribution of

centroid weights of term kiacross categories.

wðki; cjÞ ¼ 1 ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X i ðwðki;cYcjÞ Pðki; cjÞÞ2 r wðki;cYcjÞ Pðki; cjÞ Pðki; cjÞ ¼ wðki;cYcjÞ= Xm j¼1 wðki;cYcjÞ; ð7Þ

where m is the number of categories. The denominator on the right-hand side of Eq. (7) normalizes the weight of

a term. P(ki, cj) indicates the discriminating power of

term ki, i.e., its ability to distinguish between categories.

4.2.2. Deriving the degree of relevance of executed tasks to categories

We can derive the relationship (degree of relevance) between categories and executed tasks based on the cosine measure described in Section 3.1. The relevance degree of task trto category cj,μcj(tr), can be calculated

as the similarity between two vectors, Ytrand Ycj, namely,

cosineðYtr; YcjÞ. The relevance degree between a task and

a category indicates how well the task fits the category.

The relevance degrees of task trto the m categories can

be modeled as a vectortYc

rexpressed in Eq. (8).

Y

tc

r ¼ hlc1ðtrÞ; lc2ðtrÞ; N ; lcmðtrÞi ð8Þ

The task categorization database records the

catego-rization results. The association between a task trand a

category is indicated by its relevance degree to that category. The task categorization database supports the proposed two-phase task-assessment approach. Details are given in Section 5.1.

5. Task relevance-assessment and knowledge retrieval The proposed mechanism generates the task profile based on the corpus of previously executed tasks and their relevance to the current task, as evaluated by knowledge workers. Our task-based knowledge support (K-support) system provides a task-based workplace that facilitates collaborative assessment by workers. Colla-borative relevancy-assessment means that workers and task experts can conduct relevance assessment (i.e., by giving linguistic ratings to categories or tasks) of the current task in the task-based workplace of the K-support system, as shown in Appendix B. The assessment results of the evaluators (e.g., executors, colleagues, or experts)

(9)

are aggregated to derive the relevance of executed tasks to the current task, and integrated with the modified relevance feedback technique to derive the task profile.

Section 5.1 presents the proposed collaborative two-phase relevance assessment approach, while Section 5.2 describes the modified RF technique that incorporates the aggregated relevance ratings from user assessments to generate and adjust task profiles. The retrieval of knowledge items based on task profiles is described in Section 5.3.

5.1. Two-phase relevance assessment based on the fuzzy linguistic approach

A novel two-phase assessment approach is used to model the relevance assessment procedure used in the collaborative workplace. The approach reduces the number of tasks to be assessed by extracting a set of reference tasks from the task database to help workers conduct task-relevance assessment. The fuzzy linguistic approach is used to assess the degree of relevance of tasks and categories.

5.1.1. Phase 1: Identifying reference tasks based on category assessment

Phase 1 of the assessment determines the degree of relevance between the current task and the categories. The reference tasks are then identified by calculating the similarity measures based on the degree of relevance of the task to the categories.

5.1.1.1. Step 1: Determining the semantic term set and

corresponding fuzzy number. To model the workers'

perceptions of Relevance, the system defines six

linguistic terms, from “very low”, “low”, “normal”,

“high”, “very high”, “perfect” to represent the degrees of relevance. Each worker has his/her own perception of the approximate value (fuzzy scale) of each linguistic term. The fuzzy scale of a linguistic term is often modeled as a triangular fuzzy number (l, m, r), as described in Section 3.3. The linguistic terms are used in the front-end of the system to provide knowledge workers a more natural and easier way to assess relevance, while fuzzy numbers are used in the back-end to compute relevance ratings. Clearly, evaluators may not have identical fuzzy numbers

for the six linguistic terms of“Relevance”. For example,

evaluator E1's perception of“very high” is (0.6, 0.7, 0.8)

on the fuzzy scale, but evaluator E2's perception of“very

high” is (0.6, 0.75, 0.9). Each evaluator can use the front-end interface to select a fuzzy number for each linguistic term easily, or the default fuzzy number provided by the system can be used instead.

5.1.1.2. Step 2: Collaborative assessment of the relevance

of tasks to categories. This step assesses the relevance

of the current-task to each category. The executor, namely the knowledge worker responsible for the current task, rates the relevance of the task to each category by linguistic terms. Hereafter, linguistic ratings denote the ratings given to linguistic terms. In addition, task experts or colleagues can rate the relevance of the current task to each category by linguistic terms to achieve collaborative assessment through the collaborative workplace of the system. In collaborative assessment, a rating derived by aggregating the ratings of the task experts or colleagues is especially useful for a worker who is unfamiliar with the current task. However, the linguistic ratings cannot be used by the system to calculate aggregate ratings, and must therefore be transformed into crisp ratings. Linguis-tic ratings are transformed into crisp ratings in the back-end of the system. An example of relevance assessment is shown in Appendix B. Evaluators determine the degree of

relevance of the current task te to each category using

linguistic ratings. The corresponding fuzzy number of each linguistic rating is transformed into a crisp number (rating) by the center-of-area method described in Appendix A. For example, an evaluator's perception of

the linguistic term “very high” is (0.6, 0.7, 0.8) on the

fuzzy scale. The fuzzy number is transformed into a crisp value, 0.7.

5.1.1.3. Step 3: Aggregating the relevance ratings of

evaluators. Evaluators' crisp ratings obtained through

collaborative assessment are aggregated in this step. The degree of relevance of the current task to each category is derived by computing the weighted average of the evaluators' crisp ratings for the relevance of the task to the categories. The aggregated relevance of the current task to the categories is expressed as a vector of degrees

of relevance to each category. Let Aej(ci) denote the crisp

rating of evaluator ejfor the relevance of the current task

te to category ci Also, let wej denote the associated

weight, which represents the relative importance

(weight) of the rating of evaluator ej. The aggregated

relevance of the current task to category ci, AE(ci), is∑j

wejAej(ci). The degree of relevance of task te to the

categories can be modeled as a vector Ytc

e¼ hAEðc1Þ;

AEðc2Þ; N ; AEðcmÞi. If wej= 1 / ne, where nedenotes the

number of evaluators, then the aggregated relevance ratings are calculated as the arithmetic mean.

5.1.1.4. Step 4: Selecting reference tasks. This step

identifies a subset of previously executed tasks as reference tasks based on their similarity to the current task. The degree of relevance of the current task to the

(10)

categories is derived by Step 3 of the category assessment procedure, while that of an executed task is derived as described in Section 4.2. A similarity (cosine) measure is adopted to calculate the similarity between the current task and an executed task according to their degrees of relevance to the categories. Based on the similarity measures, the top-N similar executed tasks are chosen as the positive (relevant) reference tasks, and the last-M non-similar executed tasks are chosen as the negative (irrelevant) reference tasks. The reference tasks are used for further task-relevance assessment in phase 2.

The similarity between the current task te and an

executed task tr can be computed as the cosine of the

angle between two vectors, Ytc

e and Ytcr, namely, cosine

ðtYc

e;YtcrÞ; tYce is derived by the collaborative relevance

assessment described in Step 3, whileYtc

ris derived by the

categorization model described in Section 4.2.

5.1.2. Phase 2: Assessing the relevance of reference tasks Phase 2 assesses the relevance of the reference tasks to the current task. The evaluators assess the degree of relevance between the current task and the reference tasks without reviewing all tasks. The task assessment procedure is similar to that of category assessment. The evaluators use linguistic terms to assess the degree of relevance of each reference task to the current task. The aggregated relevance rating of a reference task is derived by computing the weighted average of the evaluators' crisp ratings for the relevance of the reference task to the current task. The degrees of relevance of the reference tasks to the current task are then used to construct the task profile of the current task, as described in Section 5.2.

Let Aej(tr) represent the crisp rating of evaluator ejfor

the relevance of a reference task tr to the current-task.

Also, let wej denote the associated weight representing

the relative importance (weight) of the rating of evaluator

ej. The aggregated relevance rating of task tr to the

current-task, AE(tr) is∑jwejAej(tr).

5.2. Constructing the task profile based on relevance feedback

The task profile of the current task is initially derived by analyzing the task contents (textual descriptions), or alternatively by using the corresponding task corpus. However, the initial task profile may not properly represent the current task. Collaborative task-assessment identifies the degrees relevance of the reference tasks to the current task. The result is used to refine the initial task profile based on the relevance feedback (RF) techniques introduced in Section 3.2.

Two kinds of relevance judgments about reference tasks are considered: positive feedback and negative feedback. The standard RF technique employs binary feedback without considering the degrees of relevance, as shown in Eq. (9). Relevant tasks with positive feedback have a positive influence on the weights of terms, while irrelevant tasks with negative feedback have a negative influence on the weights of terms. A refined task profile can be generated by adding the term weights of relevant tasks and subtracting the term weights of irrelevant tasks. Consequently, the feature vector of new term weights based on the RF technique forms a new task profile for further knowledge retrieval. Relevance feedback shifts the new profile closer to the relevant task set and away from the irrelevant task set. The parameters β and γ, respectively, are used to determine the relative influence of the relevant task set compared to the irrelevant task set.

We modify the Rocchio and Ide_Dec_Hi methods by considering the degrees of relevance of reference tasks obtained from the aggregated relevance ratings of fuzzy linguistic assessment. The modification considers the relative importance of relevant and irrelevant tasks from the user's perspective. The feature vectors of reference tasks are multiplied by their relevance degrees to reflect their relative contributions to the refinement of the task profile, as expressed in Eq. (10).

Based on above discussions, two RF approaches are used to construct the task profile YSeof the current task te. The RF with binary relevance assessment, denoted as B-RA (Eq. (9)), considers binary (relevant and irrelevant) assessment. The RF with fuzzy linguistic relevance assessment, denoted as F-RA (Eq. (10)), considers the degrees of relevance based on user perceptions.

B RA : YSe¼ aYSinitialþ b X 8tja Tr Y_t j g X 8tja Tn Y_t j ð9Þ F RA : YSe¼ aYSinitialþ b X 8tja Tr ðwtjÞ Y_t j g X 8tja Tn ð1 wtjÞ Y_t j; ð10Þ

where YSinitial represents the initial profile derived by

analyzing the relevant documents, if available, for the

current task; and. Tr denotes the set of relevant tasks

selected from the positive reference tasks according to the

collaborative assessment of experts and workers. Tn

represents the set of the last-M irrelevant tasks, which

the system selects automatically; Ytjis the task corpus of

a reference task tj with an associated weight wtj

(11)

AE(tj), which is the aggregated relevance rating of task tj to the current task; AE(tj) is derived by the task assessment

procedure described in Section 5.1.2; andα, β, and γ are

tuning constants.

The task profile of the current task te, derived from

Eq. (9) or (10) can be expressed as a feature vector of weighted terms, YSe¼ hwðk1; teÞ; wðk2; teÞ; N ; wðkn; teÞi,

where w(ki, te) is the weight of a term kirepresenting the

main concept terms of te; and n denotes the number of

discriminating terms. YSe is used to retrieve relevant

codified knowledge.

5.3. Task-based knowledge retrieval

A task-based knowledge support system can be realized with the proposed systematic profile modeling approach. The generated task profile is the system kernel that streamlines knowledge retrieval activity to provide task-based knowledge support. Based on task profiles, the system can recommend/retrieve relevant knowledge from the repository to assist knowledge workers. Work-ers conducting further search activity are assisted by the highly correlated term set presented in the system interface. The relevant knowledge includes relevant tasks, associated peer groups, relevant documents, and highly correlated term sets.

The similarity between the current task and the codified knowledge items can be calculated to select the top-N relevant tasks or documents from the knowledge repository. The cosine measure of feature vectors, described in Section 3.1, can be used to derive the similarity measure. The task profile can be further adjusted during the task's execution by monitoring the

workers' feedback. Our recent paper[25] presented an

adaptive task-based profiling approach for modeling workers' dynamic task needs. The codified knowledge that is relevant to the current task can be retrieved based on the adjusted task profile to fit the worker's dynamic information needs. Moreover, knowledge sharing among peer group members with similar interests is important in deploying a KMS. Task-based peer-group members with similar task needs can be identified from the retrieved relevant task set to provide knowledge sharing. The method for identifying task-based peer groups that can

support knowledge sharing is also presented in[25].

5.3.1. Relevant tasks and peer group recommendations As the task profile has been derived, retrieving relevant tasks for references would be helpful. The cosine measure is calculated to derive the similarity between the current task and an executed task. The tasks with the top-N similarity measures are recommended as relevant tasks.

These tasks and the knowledge workers engaged in them are recommended for consultation. Effectively codifying tacit knowledge may be difficult. However, the system can locate valuable knowledge sources, such as knowl-edge workers engaged in relevant tasks, thereby provid-ing a knowledge support platform for gatherprovid-ing and exchanging task-relevant knowledge among workers. 5.3.2. Relevant documents and term recommendation

Relevant documents are retrieved using the profile of the current-task. Similarity measurement is also used to select the top-N relevant documents. Documents with top-N similarity measures are selected as the relevant documents for recommendation. Meanwhile, the impor-tant term set representing the main concept terms of the current task is derived from the constructed task profile. The system displays the discriminating terms and their associated weights to assist knowledge workers with further retrieval. The term set forms the task corpus of the current task, and can be modified during subsequent stages of the task's execution.

6. Experiment evaluations

Three experiments were performed to evaluate the effectiveness of assessment and retrieval based on the proposed methods. Section 6.1 describes the experiment setup, including the experiments' objectives, data, evaluation metrics, and related parameter selection. Section 6.2 presents the experiment results.

6.1. Experiment setup 6.1.1. Experiment objectives

We conducted the following three experiments to evaluate the effectiveness of the proposed collaborative relevance-assessment approach: (1) Experiment one evaluated whether building task profiles based on binary or fuzzy linguistic relevance assessment method could help knowledge workers retrieve task-relevant informa-tion more precisely than the query-based method. The latter method, which simply employs traditional keyword search to access knowledge items without profile generation, is a user-driven approach that enables knowledge workers to express their information needs as queries to search for knowledge items. The experiment also evaluated the effectiveness of fuzzy linguistic assessment for two worker groups: experienced work-ers and novices. (2) Experiment two evaluated whether the proposed two-phase relevance assessment approach can reduce the assessment load created by a large number of tasks. (3) Experiment three evaluated

(12)

whether the proposed collaborative relevance-assess-ment method helps knowledge workers find task-relevant information more precisely with the aid of domain experts.

6.1.2. Data and participants

Experiments were performed using a real application domain, namely, research tasks in the laboratory of a research institute. The tasks consisted of writing research papers or conducting research projects. Using a real application domain restricts the sample size of the data and the number of participants in the experiments. Fifty research tasks were studied: 31 executed tasks and 19 current tasks; and eighteen workers were selected to participate in the experiments. Over 500 documents accessed by the tasks were collected. Information extraction and document pre-processing (e.g., case folding, stemming, and stop word removal) identified an average of 90 distinct terms in each document. The feature vectors of the documents were derived by the method described in Section 3.1. Historical executed tasks were categorized into five categories defined according to the ACM Computing Classification Systems.

Knowledge workers usually require a substantial amount of time (e.g., 1 year) to accomplish knowledge-intensive tasks. However, when the task performance process spans a long period, it is difficult to design experiments relevant to real-world problems. Thus, we chose evaluators according to whether they were familiar or unfamiliar with the current task's execution. Conse-quently, two worker groups were chosen: experienced workers familiar with the current task, and novices unfamiliar with the current task.

Six current tasks were chosen as the test set for the evaluations. To determine the effectiveness of collabora-tive relevance-assessment, current tasks in the test set were those in which more than one knowledge worker participated. We also chose current tasks conducted by at least one novice and one experienced worker to evaluate the effectiveness of the proposed methods for different worker groups. We selected one or two experienced workers and one or two novices from each test task as participants in the test set. The limitation of the test set selection for the problem domain restricted the test set size. 6.1.3. Performance evaluation metrics

Experiments were conducted to evaluate the effec-tiveness of our approach for retrieving knowledge items. We adopted the evaluation methodology used in Information Retrieval (IR). The IR evaluation method-ology concentrates on the evaluation of quantitative or

qualitative data [7]. Retrieval effectiveness is the most

commonly used criterion for quantitative evaluation. Qualitative evaluation of an IR system can be conducted based on various criteria, such as user satisfaction, usability and learning ability through the analysis of questionnaires. Our evaluation focused on the effective-ness of retrieval. Thus, various approaches are compared and discussed according to their performance in terms of the evaluation criteria of retrieval effectiveness. Preci-sion and recall are commonly used evaluation metrics to

measure the effectiveness of information retrieval[33].

Precision is the fraction of retrieved items (tasks or documents) that are relevant, while recall is the fraction of total known relevant items that are retrieved, defined as Eqs. (11) and (12).

precision¼jretrieved items that are relevantj

jtotal retrieved itemsj ð11Þ

recall¼jrelevant items that are retrievedj

jtotal known relevant itemsj ð12Þ

Both the total number of retrieved items and the total number of known relevant items must be greater than zero. Increasing the number of retrieved items tends to reduce precision and increase recall. Generally, precision is high at low recall levels and low at high recall levels. Thus, a recall-precision curve can be used to show the

trade-off between precision and recall [4,36]. We

evaluate the effectiveness of the proposed knowledge retrieval approach based on the recall-precision curves, which treat precision as a function of recall. The recall-precision curve plots the interpolated recall-precision at each

recall level, as follows [4,36]. The recall values are

divided into different recall levels with rvi, i∈{1,2,…, n}

denoting a reference point at the i-th recall level. The

interpolated precision, IPr(rvi) can thus be expressed as

IPr(rvi) = MAX Pr(rv) for rvi≤rvbrvi+1, where Pr(rv)

represents the precision value given a recall value of rv. The interpolated precision of each recall level can be derived for each task. The average interpolated precision for evaluating a set of tasks is derived by Eq. (13). aveIPrðrviÞ ¼

Xk

j¼1

IP_rjðrviÞ

k ð13Þ

where aveIP(rvi) denotes the average interpolated

precision at the i-th recall level; k denotes the number

of evaluated tasks; and IPrj(rvi) denotes the interpolated

precision of task j. 6.1.4. Parameter selection

We adopt and modify the classical relevance feedback methods to design the proposed relevance feedback

(13)

a pilot experiment to determine the parameters of the two classical relevance feedback methods. Their results

suggest that settingα=1, β=0.75, and γ=0.25 can

im-prove retrieval performance (i.e., a higher precision value). This work uses a similar approach to that suggested by Salton and Buckley to determine the parameter settings.

We conducted a pilot experiment to determine the

parameter values ofα, β and γ in Eqs. (9) and (10). To

adjust the relative importance of relevant and irrelevant

tasks, we setα=1 and β+γ=1. Accordingly, only one

parameter had to be determined (β or γ). The experiment

was conducted by systematically adjusting the value ofβ

in increments of 0.1. The precision metric (given in Eq. (11)) was chosen as the performance measure to evaluate the effectiveness of the methods. The optimal parameter values with the best results (the highest precision values) were chosen as the parameter settings of the proposed equations. The experiment results suggest that the best result can be achieved by setting α=1, β=0.8, and γ=0.2. This finding agrees with the conclusion of most previous studies that the information in relevant documents is more important than that in irrelevant documents. Thus, the above parameter settings were adopted in our experiments.

6.2. Experiment results

Experiment one compares the binary relevance assessment method (B-RA method) and the fuzzy linguistic relevance assessment method (F-RA method) with the query-based method. B-RA and F-RA are one-phase relevance assessment methods that only conduct task-relevance assessment (phase 2), as described in Section 5.2, without employing phase-1 category-rele-vance assessment and collaborative assessment. Experi-ment two measures the impact of the assessExperi-ment load while conducting task-relevance assessment. According-ly, the two-phase relevance assessment approach (denoted as 2-F-RA) is compared with the one-phase relevance assessment approach, F-RA. The 2-F-RA method con-ducts both the phase-1 category relevance assessment and the phase-2 task-relevance assessment without employing collaborative assessment. The phase-1 assessment deter-mines the relevance of the current task to the categories, and then identifies the reference tasks by computing the similarity measures based on the degree of relevance of the tasks to the categories. The third experiment evaluates the effectiveness of collaborative two-phase assessment (denoted as Collaborative 2-F-RA) versus non-collabora-tive two-phase assessment (2-F-RA). Collaboranon-collabora-tive as-sessment aggregates the relevance ratings derived from the assessments of experts and collaborative workers.

6.2.1. Experiment one: effect of fuzzy linguistic assessment This experiment evaluates the effectiveness of finding task-relevant information by the query-based method, the B-RA method and the F-RA method. The B-RA method employs binary (relevant and irrelevant) assess-ment and relevance feedback without considering the degree of relevance. The F-RA method considers the degree of relevance in the assessment and in relevance feedback, i.e., it models the user's perception value by the fuzzy linguistic rating approach. Recall that the B-RA and F-RA methods conduct task-relevance assess-ment (phase 2) without employing phase-1 category-relevance assessment or collaborative assessment.

As noted earlier, six current tasks were chosen as test tasks. The experiment employed two worker groups:

experienced workers and novices. Table 1 shows the

effectiveness of knowledge support for task-retrieval by listing the average interpolated precision of each recall level, computed over the test tasks, for the three methods and the two worker groups. The recall level [rvi, rvi+1) denotes the interval of recall values that

satisfy rvi≤recallbrvi+1. The last row shows the average

precision values computed over all recall levels.

6.2.1.1. Observations and implications. The average

precision values of the B-RA and F-RA methods exceed those of the query-based method for both experienced workers and novices. The results show that building task profiles by assessing the relevance of previously executed tasks can help knowledge workers retrieve task-relevant information. For experienced knowledge workers, the average precision of F-RA is higher than that of B-RA. This result indicates that the F-RA method provides better knowledge support to experienced workers than the B-RA method.

Fig. 2plots the average recall-precision curves of the three proposed methods, and shows a gradual decrease in the average precision value. E_F-RA and E_B-RA denote experienced workers, while and N_F-RA and

N_B-RA denote“novices”. The experiment results show

that the average precision values of F-RA and B-RA for experienced workers exceed those of F-RA and B-RA for novices. Thus, the proposed assessment approach provides experienced workers with better knowledge support than novices.

Interestingly, the average precision values of B-RA

and F-RA for novices are similar. Table 2 lists the

average precision values of task retrieval for ten novices. In three cases, the average precision value of F-RA is lower than that of B-RA. We observe that some novices cannot obtain better knowledge support from F-RA than they obtain from B-RA. This result implies that

(14)

experienced workers are knowledgeable about making appropriate assessments regarding the degree of rele-vance using the fuzzy linguistic approach. However, some novices do not have sufficient knowledge to determine the degree of relevance of the current task and previously executed tasks. Therefore, a simple binary assessment (relevant or irrelevant) may be more appro-priate for novices.

Table 3 shows the results of knowledge support for document-retrieval based on the B-RA and F-RA methods. The observations and implications of docu-ment retrieval are similar to those of task retrieval. 6.2.2. Experiment two: effect of two-phase relevance assessment

This experiment evaluates whether reducing the number of tasks for assessment can help workers conduct task relevance assessment. The effectiveness of knowl-edge support is evaluated by two-phase fuzzy linguistic relevance assessment (2-F-RA) and one-phase fuzzy linguistic relevance assessment (F-RA). Experiment one demonstrates that the average precision values of F-RA method exceed those of the B-RA and query-based methods, especially for the experienced workers. There-fore, we chose the F-RA method instead of the B-RA method or query-based method for comparison with the 2-F-RA method. The 2-2-F-RA approach reduces the number

of tasks by selecting reference tasks based on category assessment (phase 1) described in Section 5.1.1. The one-phase F-RA approach conducts task relevance assessment (phase 2) without performing phase-1 assessment.

Tables 4 and 5show the effectiveness of task-retrieval and document-retrieval, using the F-RA and 2-F-RA

methods respectively.Fig. 3plots the average

recall-pre-cision curves of two assessment methods based onTable 4.

6.2.2.1. Observations and implications. The results

show that the overall average precision using the 2-F-RA method is higher than that of F-RA method for both worker groups. This implies that two-phase relevance assessment (2-F-RA) provides better knowledge support for task and document retrieval than the F-RA method. Because category assessment (phase 1) reduces the burden of assessing the relevance of a large number of tasks, two-phase assessment can help workers conduct task-relevance assessment more effectively than one-phase assessment.

Fig. 3shows that knowledge support is more effective for experienced workers than for novices. Experienced workers (E_2-F-RA, E_F-RA) are more knowledgeable and thus derive more effective knowledge support than

novices (N_2-F-RA, N_F-RA). However, inFig. 3, the

curves for novices cross at some points. We made a further check of each case and found that, in two cases, Table 1

Result of knowledge support for task retrieval (B-RA versus F-RA)

Recall level Experience users Novices

Query B-RA F-RA Query B-RA F-RA

Precision Precision Precision Precision Precision Precision

[0.0, 0.2) 1.000 1.000 1.000 1.000 1.000 1.000 [0.2, 0.4) 0.745 0.945 0.944 0.670 0.762 0.762 [0.4, 0.6) 0.645 0.833 0.889 0.566 0.648 0.644 [0.6, 0.8) 0.502 0.733 0.820 0.458 0.610 0.600 [0.8, 1.0) 0.497 0.616 0.623 0.403 0.402 0.407 [1.0, 1.0] 0.333 0.395 0.396 0.359 0.331 0.351 6-pt average precision 0.620 0.754 0.779 0.576 0.626 0.627

The data in bold indicates that the value of the method is higher than that of the other method in the comparison.

(15)

the average precision value of 2-F-RA was lower than that of F-RA. In other words, some novices could not obtain better knowledge support from two-phase assess-ment than from one-phase assessassess-ment. On the other hand, the average precision value of 2-F-RA was higher than that of F-RA for each case of experienced workers. The result implies that, for some novices, one method does not help them more than the other in the retrieval of task-relevant knowledge. This is because they are unfamiliar with their tasks and therefore find it difficult to perform task-relevant assessments. Thus, some nov-ices may require assistance with the assessment task. 6.2.3. Experiment three: effect of collaborative assessment

The objective of this experiment is to show that collaborative assessment reduces the workload of novices and helps them find task-relevant information. Novices who are less knowledgeable about a task in the initial stages may have difficulty in performing task-relevance assessment. The effect of collaborative assessment is compared with that of non-collaborative assessment. Experiment two showed that two-phase relevance assessment (2-F-RA) provides better knowledge support for task retrieval and document retrieval than the F-RA method. This experiment goes a step further and evaluates the effectiveness of collaborative relevance assessment. Collaborative assessment aggregates the relevance ratings derived from the assessment of experienced workers and novices, as discussed in Section 5.1.2. The individual

assessments of novices are considered as non-collabora-tive assessments to derive the ratings of task relevance.

Table 6shows the effectiveness of knowledge support for task retrieval and document retrieval under collab-orative 2-F-RA (by Experienced workers and Novices) and non-collaborative 2-F-RA (by novices). The com-parison is based the average interpolated precision at six recall levels and their aggregated average.

6.2.3.1. Observations and implications. The results

show that the collaborative 2-F-RA method is more effective than the non-collaborative 2-F-RA method. Thus, novices can obtain more effective knowledge support through collaboration with experienced workers by adopting the collaborative relevance-assessment approach. Collaboration among knowledge workers can also mitigate the difficulty of retrieving task-relevant knowledge from the knowledge repository.

6.2.4. Discussion

Experiments were conducted to evaluate the effective-ness of the proposed collaborative relevance-assessment approach (Collaborative 2-F-RA). In all, we conducted three experiments to assess the effect of linguistic assessment, two-phase relevance assessment, and collab-orative assessment respectively. The results demonstrate that: (1) linguistic relevance assessment (F-RA) provides better knowledge support than binary assessment (B-RA) or the query-based method; (2) two-phase relevance assessment (2-F-RA) provides better knowledge support Table 2

Results of knowledge support for task retrieval by ten novices

N1(T1) N2(T2) N3T(2) N4(T3) N5(T4) N6(T4) N7(T5) N8(T5) N9(T6) N10(T6)

B-RA 0.778 0.549 0.643 0.691 0.814 0.577 0.450 0.644 0.571 0.462

F-RA 0.786 0.559 0.643 0.691 0.805 0.552 0.467 0.647 0.508 0.475

Table 3

Results of knowledge support for document retrieval (B-RA versus F-RA)

Recall level Experience workers Novices

B-RA F-RA B-RA F-RA

Precision Precision Precision Precision

[0.0, 0.2) 0.650 0.650 0.669 0.703 [0.2, 0.4) 0.280 0.306 0.184 0.203 [0.4, 0.6) 0.238 0.271 0.172 0.189 [0.6, 0.8) 0.202 0.227 0.155 0.168 [0.8, 1.0) 0.150 0.168 0.140 0.152 [1.0, 1.0] 0.129 0.144 0.129 0.134 6-pt average precision 0.275 0.294 0.242 0.258

Table 4

Knowledge support for task-retrieval (2-F-RA versus F-RA)

F-RA 2-F-RA F-RA 2-F-RA

(16)

than one-phase relevance assessment (F-RA); and (3) collaborative relevance assessment (Collaborative 2-F-RA) provides better knowledge support than non-collaborative relevance assessment (2-F-RA). Although the improvement achieved by adding one more factor is not significant, the improvement of the collaborative 2-F-RA method over the query-based method is significant. The results of the query-based method (baseline method)

are listed inTable 1, and the results of the collaborative

relevance assessment method (Collaborative 2-F-RA) are

listed in Table 6. For novices, collaborative 2-F-RA is

25.86% more effective than the query-based method. The results demonstrate that the proposed collaborative relevance-assessment approach can provide effective knowledge support in task-based environments.

As one would expect, experienced workers are more knowledgeable about relevance assessment than novices, and can therefore make appropriate assessments using the fuzzy linguistic approach. In contrast, a simple binary assessment (relevant or irrelevant) approach may be more appropriate for some novices when they are unfamiliar with a task. Novices may also benefit from collaboration with experienced workers when conduct-ing task relevance-assessment.

Our results indicate that two-phase assessment (incor-porated with category assessment) can help workers con-duct task-relevance assessment more effectively than

one-phase assessment. In our preliminary experiment, we asked the subjects to evaluate the tasks (i.e., to indicate the degree of relevance by a fuzzy linguistic rating). Unfortunately, we found the assessment was very time-consuming because there were too many task items (fifty-one in our application domain) that needed to be browsed or read in advance. Thus, we sought to classify the tasks into five categories which were predefined by experts to generalize our application domain. In our experiment, we found that two-phase assessment is more effective than one-phase assessment for both types of worker groups, but especially for experienced workers, who can make better assessments than novices. The experiment results suggest that multi-level categorization can further reduce the assessment load. However, we also observed that some novices were confused about the categories. Novices may have difficulty in recognizing the differences between categories and understanding the topic of each category. This observation implies that increasing the levels of categorization structure may also increase a worker's assessment burden.

Generally, if the number of categories is large, multi-level categorization may further reduce the burden of category assessment. Based on our observations of the experiments, we address the tradeoff and justification for Table 5

Knowledge support for document retrieval (2-F-RA versus F-RA)

F-RA 2-F-RA F-RA 2-F-RA

Fig. 3. Average recall-precision curves for task retrieval by experienced workers and novices. Table 6

Results of knowledge support (non-collaborative 2-F-RA versus collaborative 2-F-RA)

Recall level Task retrieval Document retrieval

Non-C. 2-F-RA (Novice) Colla. 2-F-RA (E and N) Non-C. 2-F-RA (Novice) Colla. 2-F-RA (E and N) Precision Precision Precision Precision