6. Experiments
6.2. Experimental Result
6.2.1. Experiment one: effect on task-stage knowledge support with different size of time window
(1) Objective
The objective of this experiment is to evaluate the effectiveness of our proposed method with different time window size. As we have mentioned in Section 4.2, a time window size w is set to define the scope of candidate clusters to be merged. The smaller the time window size is, the higher the timeliness is. When time window size is equal to n (the number of documents in TDS of a task instance), the proposed TSHC algorithm is the same as standard hierarchical clustering algorithm (HAC). Thus, we conduct experiment to evaluate the impact of the time window parameter. And then the best time window size w will be selected for the usage of the proposed stage knowledge support method.
(2) Experimental Result and Observations
Table 9 shows the experimental results with different time window size in terms of precision, recall, and F1-measure. Four window sizes are evaluated: w=1, w=2, w=3 and w=n.
Three virtual tasks are evaluated. In addition, we also evaluated the effectiveness of the proposed method under various top-N (N=10, 20, and 30) knowledge support, i.e., providing top-N documents with N-highest ranked similarity measures.
Observation 1: Generally, the best performance can be achieved when time window equals to one (w=1). Bold type denotes the best performance with different time window size.
Specifically, when conducting top-30 knowledge support, time window equals to one (w=1) has the best performance. In addition, it is also apparently that the greater the time window, the lower effectiveness of the proposed method.
not always leading to best performance for virtual task 1. It indicates that higher variant of TDS (task document sequence) exists for task instances of virtual task 1.
Table 9. Result of knowledge support with different time window for three virtual tasks
Parameters Virtual Task 1 Virtual Task 2 Virtual Task 3 Top-N w Pre. Re. F1-measure Pre. Re. F1-measure Pre. Re. F1-measure
w=1 0.267 0.203 0.231 0.231 0.084 0.171 0.344 0.175 0.232
w=2 0.255 0.161 0.197 0.166 0.065 0.127 0.344 0.175 0.232
w=3 None None None 0.156 0.063 0.121 0.244 0.124 0.165 Top-30
w=n 0.167 0.116 0.137 0.133 0.019 0.033 None None None
w=1 0.267 0.126 0.171 0.217 0.069 0.152 0.350 0.118 0.177
w=2 0.350 0.152 0.212 0.156 0.049 0.118 0.350 0.118 0.177 w=3 None None None 0.133 0.037 0.088 0.250 0.085 0.126 Top-20
w=n 0.167 0.079 0.107 0.150 0.039 0.062 None None None w=1 0.333 0.080 0.129 0.300 0.047 0.144 0.433 0.074 0.126
w=2 0.367 0.080 0.132 0.200 0.026 0.085 0.433 0.074 0.126 w=3 None None None 0.167 0.023 0.074 0.333 0.056 0.096 Top-10
w=n 0.200 0.050 0.080 0.156 0.059 0.086 None None None
Note: “None” denotes the proposed method is unable to generate 3 stages (clusters)
Table 10 shows the knowledge support results with different time window size of different task stages in terms of precision, recall, and F1-measure. As we have mentioned in Section 5.1.2, due to the limitation of stage answering set, only three stages of each virtual task are identified. According to previous pilot study (Vakkari 2000, 2003), stage one indicates the task pre-focus stage; stage two indicates the task focus formulation stage; and stage three indicates the task post-focus stage. Note that the measure of each stage is the average of three virtual tasks.
Observation 3: Generally, the best performance can be achieved for all virtual task at stage 2 and stage 3 while conducting Top-20 and Top-10 knowledge support, when the time window size is set to one (w=1). Bold type denotes the best performance with different time window size. However, for the same top-N knowledge support, the best performance of different task stage is achieved in different time window size.
Observation 4: Generally, the greater the time window size, the lower effectiveness of the proposed method, especially for stage 2 and stage 3. However, stage 1 seems abnormal compared with stage 2 and stage 3. That is the greater the time window size, the higher effectiveness of the proposed method, especially in providing more documents.
Table 10. Result of knowledge support under different time window size for task stages
Parameters Stage1 Stage 2 Stage3
Top-N w Pre. Re. F-measure Pre. Re. F-measure Pre. Re. F-measure w=1 0.333 0.183 0.119 0.222 0.131 0.163 0.233 0.147 0.176
w=2 0.366 0.195 0.251 0.233 0.098 0.137 0.167 0.107 0.126 w=3 0.234 0.095 0.133 0.117 0.055 0.075 0.250 0.132 0.172 Top-30
w=n 0.367 0.253 0.200 0.084 0.034 0.067 0.033 0.029 0.018 w=1 0.333 0.122 0.177 0.283 0.112 0.159 0.217 0.080 0.116 w=2 0.433 0.249 0.179 0.283 0.080 0.125 0.167 0.060 0.088 w=3 0.225 0.064 0.098 0.125 0.040 0.060 0.117 0.055 0.075 Top-20
w=n 0.350 0.184 0.128 0.075 0.021 0.046 0.050 0.029 0.022
w=1 0.400 0.070 0.119 0.400 0.080 0.133 0.267 0.049 0.083 w=2 0.433 0.089 0.147 0.367 0.054 0.094 0.200 0.036 0.061 w=3 0.350 0.053 0.091 0.250 0.040 0.068 0.150 0.027 0.029 Top-10
w=n 0.350 0.111 0.067 0.050 0.007 0.020 0.100 0.029 0.028
(3) Implications:
This experiment evaluates the impact of the time window size in the proposed TSHC algorithm. The result reveals that generally the smaller the time window size, the better the effectiveness (the higher precision, recall, and F-measure) of knowledge support. Accordingly, we set time window size to one (w=1) in the proposed algorithm.
Interestingly, when time window size is equal to n (number of document TDS of a task instance), the proposed TSHC algorithm is the same as standard hierarchical clustering algorithm (HAC). The time effect disappeared when time window size is equal to n. The result is similar to topics clustering, i.e., documents within the same cluster may discuss a similar topic without considering the effect of stage.
6.2.2 Experiment two: comparing task-stage knowledge support with non-stage knowledge support based on overall match
(1) Objective
The objective of this experiment is to evaluate the effectiveness of task-stage mining method for knowledge support. Task-stage profiles are generated to model workers’ task-stage needs and are used for delivering task-relevant knowledge at various task stages. The task stage knowledge support method is compared with the baseline method, non-stage knowledge support method (task-based knowledge support). The task-stage knowledge support denotes providing needed documents based on task-stage profiles, whereas non-stage knowledge support denotes providing knowledge support based on task profiles. The task-stage profiles or task profiles are used to deliver task-relevant knowledge for an executing task that is similar to the virtual task. Note that this experiment compares task-stage knowledge support with non-stage knowledge support according to the task-answering set of a virtual task (named overall match). On the other hand, the experiment three compares task-stage knowledge support with non-stage knowledge support according to the stage answering set of a virtual task (named stage match).
(2) Result and Observations
Table 11 shows the result of task-stage knowledge support method and non-stage knowledge support method in terms of precision, recall, and F1-measure. Notably, three virtual tasks are evaluated under various top-N (N=10, 20, and 30) document supports. Table 12 shows the average result of task-stage knowledge support method and non-stage knowledge support method.
Observation 1: The precision, recall and F1-measure of task stage knowledge support method are greater than the non-stage method for virtual task one and two. However, for virtual task three, the precision, recall and F1-measure of task stage knowledge support method are lesser than non-stage method. The precision of non-stage method under Top-30 support is greater than that of the task stage knowledge support method.
Observation 2: Notably, for task stage knowledge support method, the fewer number of supporting documents, the higher precision value is. The rank of precision value is Top-10 >
Top-20 > Top-30. The observation doest not apply to the non-stage knowledge support method. The precision values of task stage knowledge support method are greater than those of the non-stage method under top-10 knowledge support for three virtual tasks. The result reveals that task stage knowledge support method can provide more effective knowledge
support than the non-stage knowledge support method when providing fewer number of task-stage relevant documents.
Table 11. Compare task stage knowledge support method with none-stage knowledge support method for three virtual tasks
Task Stage Knowledge Support Non-Stage (Task-based) Knowledge Support
Pre. Re. F-measure Pre. Re. F-measure
Top-10 0.667 0.082 0.146 0.600 0.078 0.138
Top-20 0.633 0.165 0.261 0.600 0.156 0.248
Top-30 0.578 0.225 0.324 0.567 0.220 0.317
Virtual Task 1
Average 0.626 0.157 0.244 0.589 0.151 0.234
Top-10 0.433 0.03 0.06 0.300 0.021 0.039
Top-20 0.350 0.05 0.09 0.400 0.056 0.098
Top-30 0.322 0.07 0.11 0.400 0.085 0.140
Virtual Task 2
Average 0.369 0.050 0.086 0.367 0.054 0.093
Top-10 0.567 0.060 0.109 0.500 0.053 0.096
Top-20 0.467 0.099 0.163 0.450 0.096 0.158
Top-30 0.444 0.142 0.215 0.567 0.181 0.274
Virtual Task 3
Average 0.493 0.100 0.162 0.506 0.110 0.176
Observation 3: Table 11 shows the details of the experiment result, which shows that virtual task one has best result. This may result from that the best clustering number for virtual task one is 3, but clustering number for virtual task two and three are not 3. As we have addressed in step 1 of Section 4.2, we use the Q value to determine the cluster quality for deciding the number of clusters of each task.
Observation 4: Table 12 shows the result of task stage knowledge support method and non-stage knowledge support method. The average precision, recall and F1-measure of task stage knowledge support method are greater than those of the non-stage method under top-10 and top-20 knowledge support. Meanwhile, for task stage knowledge support method, the fewer number of supporting documents, the higher precision value is. The rank of precision value is Top-10 > Top-20 > Top-30. For non- stage knowledge support method, the fewer number of supporting documents, the lower precision value is. The rank of precision value is Top-30 > Top-20 > Top-10.
Table 12. Results of knowledge support by overall match
Stage knowledge support Non- stage knowledge support β=1 Precision Recall F1-measure Precision Recall F1-measure
Top-10 0.556 0.058 0.104 0.467 0.051 0.091 Top-20 0.483 0.104 0.171 0.483 0.103 0.168 Top-30 0.448 0.145 0.217 0.511 0.162 0.244 Average 0.496 0.102 0.164 0.487 0.105 0.168
Table 13, and Figure 10 (A), (B), and (C) shows the results of task stage knowledge support method with none-stage knowledge support method of different task stages. Note that the none-stage knowledge support method of different task stages has the same performance value. The reason is that the non-stage knowledge support method only has one task profile.
In addition, this experiment evaluates the performance of two methods according to the task-answering set (named overall match) not the stage answering sets.
Observation 5: Generally, task stage knowledge support method has better performance than non-stage method under top-10 knowledge support for three task stages. In addition, no matter top-10, 20 or 30 knowledge supports, task stage knowledge support method has better performance than non-stage method in task pre-focus stage (stage 1).
Table 13: Compare task stage knowledge support method with none-stage knowledge support method under different task stages
Task Stage-based Knowledge Support Non-Stage (Task-based) Knowledge Support Pre. Re. F-measure Pre. Re. F-measure
Top-10 0.567 0.064 0.218 0.467 0.051 0.177
Top-20 0.517 0.120 0.309 0.483 0.103 0.278
Top-30 0.522 0.181 0.376 0.511 0.162 0.357
Stage 1 (Pre-focus)
Average 0.535 0.122 0.301 0.487 0.105 0.270
Top-10 0.500 0.051 0.179 0.467 0.051 0.177
Top-20 0.417 0.091 0.240 0.483 0.103 0.278
Top-30 0.311 0.101 0.217 0.511 0.162 0.357
Stage 2 (Focus)
Average 0.409 0.081 0.212 0.487 0.105 0.270
Top-10 0.600 0.059 0.209 0.467 0.051 0.177
Top-20 0.517 0.102 0.281 0.483 0.103 0.278
Top-30 0.511 0.153 0.343 0.511 0.162 0.357
Stage 3 (Post-focus)
Average 0.543 0.105 0.277 0.487 0.105 0.270 Note: The details of stage knowledge support of each group are listed in Appendix B1
Observation 6: Figure 10(A), (B), and (C) depict the result of knowledge support based on various number of top-N knowledge support. Interestingly, stage 1 and stage 3 of task-stage knowledge support method has better performance than non-stage knowledge support.
However, stage 2 of task-stage knowledge support method is not always better than non-stage knowledge support method under various top-N knowledge supports.
Top-10 Knowledge Support
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
Precision Recall F-measure
Stage 1 Stage 2 Stage 3 Non-stage
Figure 10(A): Result of top-10 knowledge support
Top-20 Knowledge Support
0 0.1 0.2 0.3 0.4 0.5 0.6
Precision Recall F-measure
Stage 1 Stage 2 Stage 3 Non-stage
Figure 10(B): Result of top-20 knowledge support
Top-30 Knowledge Support
0 0.1 0.2 0.3 0.4 0.5 0.6
Precision Recall F-measure
Stage 1 Stage 2 Stage 3 Non-stage
Figure 10(C): Result of top-30 knowledge support
(3) Implications:
This experiment evaluates the effectiveness of the proposed task-stage mining method for knowledge support. The experimental result reveals that generally the task stage knowledge support method is better than the baseline method, non-stage knowledge support method (task-based knowledge support). Notably, for task stage knowledge support method, the fewer number of supporting documents, the higher precision value is (Top-10 > Top-20 > Top-30).
The result revels that the proposed stage mining method can provide more effective knowledge support. Meanwhile, knowledge workers generally do not need a lot of documents at each task stage; therefore, the system needs to determine appropriate number of needed documents to support task-execution. The experimental result is useful to suggest the number of supporting documents at each task stage.
6.2.3. Experiment three: comparing task-stage knowledge support with non-stage knowledge support method based on stage match
(1) Objective
The objective of this experiment is to evaluate the effectiveness of task-stage mining method for knowledge support. Experiment three compares task stage knowledge support with non-stage knowledge support according to the stage answering set (named stage match). This experiment is similar to experiment two except that we use different answering set (stage answering set) for evaluating the effectiveness of the proposed method.
(2) Result and Observations
Table 14 shows the result of task stage knowledge support method and non-stage knowledge support method in terms of precision, recall, and F1-measure. Notably, three virtual tasks are evaluated under various top-N (N=10, 20, and 30) document supports. Table 15 shows the average result of task stage knowledge support method and non-stage knowledge support method, respectively.
Observation 1: The average precision, recall and F1-measure of task stage knowledge support method are greater than non-stage method for virtual task two and three. However, for virtual task one, the average precision, recall and F1-measure of task stage knowledge support method are lesser than the non-stage method. Only top-10 document support of task stage knowledge support method is better than the non-stage method.
Observation 2: Notably, for task stage knowledge support method, the fewer number of supporting documents, the higher precision value is. The rank of precision value is Top-10 >
Top-20 > Top-30. On the other hand, this observation does not apply to the non-stage knowledge support method. The precision values of task stage knowledge support method are greater than those of the non-stage method under top-10 knowledge support for three virtual tasks. The result reveals that task stage knowledge support method can provide more effective knowledge support than the non-stage knowledge support method when supporting fewer number of task-stage relevant documents. This result is in accordance with the result of experiment two.
Observation 3: Table 15 shows the average result of task stage knowledge support method and non-stage knowledge support method. The average precision, recall and F1-measure of
stage answering set is far fewer than the number of documents of the task answering set. This experiment uses the stage-answering sets, whereas the experiment two uses only one task answering set. The task answering set of a virtual task is the union of stage answering sets of a virtual task.
Table 14. Compare task-stage knowledge support method with none-stage knowledge support method for three virtual tasks
Task Stage Knowledge Support Non-Stage (Task-based) Knowledge Support
Pre. Re. F-measure Pre. Re. F-measure
Top-10 0.333 0.080 0.128 0.300 0.074 0.128
Top-20 0.267 0.126 0.171 0.317 0.162 0.171
Top-30 0.267 0.203 0.229 0.289 0.229 0.229
Virtual Task 1
Average 0.289 0.136 0.176 0.302 0.155 0.176
Top-10 0.300 0.047 0.080 0.233 0.033 0.133
Top-20 0.217 0.069 0.104 0.167 0.051 0.117
Top-30 0.231 0.084 0.113 0.178 0.079 0.122
Virtual Task 2
Average 0.249 0.067 0.099 0.193 0.055 0.124
Top-10 0.433 0.074 0.135 0.233 0.039 0.167
Top-20 0.350 0.118 0.194 0.300 0.102 0.200
Top-30 0.344 0.175 0.232 0.367 0.253 0.233
Virtual Task 3
Average 0.376 0.122 0.187 0.300 0.131 0.200
Table 15. Average results of knowledge support for three virtual tasks by stage match
Stage knowledge support Non- stage knowledge support
β=1 Precision Recall F1-measure Precision Recall F1-measure
Top-10 0.356 0.067 0.115 0.256 0.049 0.143
Top-20 0.278 0.105 0.156 0.261 0.105 0.163
Top-30 0.281 0.154 0.191 0.278 0.187 0.195
Average 0.305 0.108 0.154 0.265 0.114 0.167
Table 16 shows the results of task stage knowledge support method with none-stage knowledge support method under different task stages.
Observation 4: Generally, task stage knowledge support method has better performance than the non-stage method under top-10 or top-20 knowledge support for three task stages. In addition, no matter top-10, 20 or 30 knowledge supports, task stage knowledge support method has better performance than the non-stage method in task pre-focus stage (stage 1).
Table 16. Compare task stage knowledge support method with none-stage knowledge support
method under different task stages (Stage match)
Task Stage-based Knowledge Support Non-Stage (Task-based) Knowledge Support Pre. Re. F-measure Pre. Re. F-measure
Top-10 0.400 0.070 0.119 0.367 0.066 0.111
Top-20 0.333 0.122 0.177 0.333 0.126 0.181
Top-30 0.333 0.183 0.234 0.333 0.248 0.277
Stage 1 (Pre-focus)
Average 0.355 0.125 0.176 0.344 0.147 0.190
Top-10 0.400 0.080 0.133 0.233 0.043 0.072
Top-20 0.283 0.112 0.159 0.233 0.091 0.129
Top-30 0.222 0.131 0.163 0.267 0.153 0.192
Stage 2 (Focus)
Average 0.302 0.108 0.151 0.244 0.096 0.131
Top-10 0.267 0.049 0.083 0.167 0.033 0.057
Top-20 0.217 0.080 0.116 0.217 0.102 0.152
Top-30 0.233 0.147 0.176 0.234 0.186 0.247
Stage 3 (Post-focus)
Average 0.239 0.092 0.125 0.206 0.107 0.152 Note: The details of stage knowledge support of each group are listed in Appendix B2-1 and B2-2
(3) Implications:
This experiment evaluates the effectiveness of the proposed task-stage mining method for knowledge support. The results prove that the effectiveness of task stage knowledge support method by the TSHC algorithm is better than the non-stage method. This experiment has more significant improvement in precision by task stage knowledge support than that of experiment 2. Table 17 shows the comparison of Table 12 and Table 15 in experiment 2 and 3, respectively. Notably, in average, the recall value of non-stage knowledge support is better than that of the task stage knowledge support. But if we took further analysis, the recall value of top-30 task stage knowledge support is much lower than that of non-stage knowledge support. This result implies that the system can deliver more task-relevant documents by providing top-30 documents based on task profiles and providing top-10 or top-20 documents based on task-stage profiles.
Stage knowledge support Non- stage knowledge support
β=1 Precision Recall F1-measure Precision Recall F1-measure
Average results of knowledge support for three virtual tasks by stage match
Top-10 0.356 0.067 0.115 0.256 0.049 0.143
Top-20 0.278 0.105 0.156 0.261 0.105 0.163
Top-30 0.281 0.154 0.191 0.278 0.187 0.195
Average
0. 305 (+15.09%)
0. 108 (-5.26%)
0. 154
0.265 0.114 0.167
Average results of knowledge support for three virtual tasks by overall match
Top-10 0.556 0.058 0.104 0.467 0.051 0.091
Top-20 0.483 0.104 0.171 0.483 0.103 0.168
Top-30 0.448 0.145 0.217 0.511 0.162 0.244
Average 0.496 (+1.85%)
0.102
(-2.94%) 1. 0.164 0.487 0.105 0.168