Knowledge Sharing Discussion Activity for Learner Community using a Peer-Assessment Strategy

CHAPTER 3

Knowledge Sharing Discussion Activity for Learner Community using a Peer-Assessment Strategy

This sub-study designs an online PBL knowledge sharing discussion activity with

peer assessment, and conducts empirical observations. Quantitative content analysis, lag

sequence analysis and original protocol analysis are used to explore the depth of l e a r ner s ’ knowl e dge c ons t r uc t i on a nd t he pa t t e r n of di s c us s i on be ha vi or a mong

members of the learner community. The study explores the influence and limitations of

the activity with regards to knowledge interaction; suggestions can be discussed

accordingly for teachers and technology developer.

3.1 Discussion Activity

Through peer review and discussion, peer assessment promotes critical thinking

and meta-cognitive skills (Topping, 1998). To design an online PBL knowledge

sharing mechanism that combines peer-assessment strategies, this study compiled

literature related to peer-assessment and integrated knowledge sharing and PBL.

Various online discussion behaviors in the mechanism may help to assist knowledge

internalization and externalization. The activity is described as follows:

Step1: Project report writing

First, the teacher asked each student to complete a project report. The

students had to choose their own project topic and use the internet to gather,

organize, and analyze information; the report was completed according to a format

provided by the teacher, where the report had to be completed independently before

a given deadline.

Step2: Upload the report

After the due date for report submission, all students had to upload the report

file onto an online learning system provided by the teacher so the files could be

downloaded and shared with their peers. The system here must provide relevant

functions that are available for online PBL; in addition to uploading a report and br ows i ng pe er s ’ r e por t s , a n a s ync hr onous f or um s houl d be pr ovi de d f or pee r

assessment discussions. The forum allowed the students to post new discussion

topics, with each discussion topic available for others to read and post replies.

Step3: Post an introduction topic

After the project reports were completed, each student was required to post a

new discussion topic on the online forum to introduce their work. All students

therefore had topics about their own report; in the consequent peer assessment,

other peers would enter the topic and comment by posting relevant messages.

Step4: Perform peer assessment

Fi na l l y , t he t e a c he r woul d a l l ow s t ude nt s t o downl oa d a nd vi e w t he i r pe er s ’

project reports after all students posted their report introduction topics. The teacher

asked the students to go into the forum, browse the list of topics and read the

introductions written by their peers. Within certain time limits, the students were

encoura ge d t o a s s e s s t he i r pe e r s ’ r e por t s by pos t i ng que s t i ons or c omme nt s a nd

reply to questions or comments from peers directed at their own topics.

The relationships between the above activity and how it assists knowledge

sharing are shown in Table 1. The table describes various online discussion behaviors

in the activity and the scope for knowledge sharing of these behaviors, which can

assist knowledge internalization and externalization.

Table 1. An online knowledge sharing discussion activity with peer assessment Behavior Description How it helps knowledge

internalization

How it helps knowledge externalization

Write project introductions and review pe e r s ’ wor k

Each learner posts an article to introduce and describe the connotation of his own work. At the same time, the learner browses ot he r pe er s ’ wor k and their

introductions.

Read, understand and internalize the content of one ’ s wor k, a nd t hi nk about how to write an abstract to describe the connotation of the work.

Externalize the abstract of one ’ s wor k via text and share with other members.

Propose questions or comments to peers

Post articles; raise questions or comments to peers.

Read, understand and internalize the works of peers and their posted introductions.

Externalize questions or comments via text to ot he r ’ s wor ks a nd s ha r e with other members.

Reply to pe e r s ’ questions or comments

Reply to questions raised by others by posting articles to answer or discuss.

Read, understand and internalize knowledge related to content of questions raised by other members, and think about how to reply accordingly.

Ext e r na l i ze one ’ s r e pl y in text towards opinions of peers, and share such replies with other members.

As described in Table 1, the content and aspect of each online discussion

behavior to assist knowledge internalization (i.e.: understand, read and think) and

knowledge externalization (i.e.: write and demonstrate) differs. This study provided

an online discussion environment for students to carry out discussions. Content

analysis and sequential analysis were then implemented on discussion records, so the

pattern and content of peer assessment discussion could be further explored.

3.2 Methods

3.2.1 Participants

The 45 participants in this study were senior college students majoring in

Information Management. The course they attended was a 3-credit course entitled

“ Comput e r As s i s t e d I ns t r uc t i on. ” The c our s e t opi c s i nc l ude d f unda me nt al t he or i e s

of e-learning, case studies, and how companies can use e-learning in their training

programs.

3.2.2 Tools

3.2.2.1 WIDE System

To implement and observe the knowledge sharing discussion activity

process of learner/teacher online discussions, an online interactive instructional

environment should be provided for online instruction and knowledge sharing.

This environment should include the online asynchronous forums that record the c ommuni t y me mbe r s ’ di s c us s i on pr oc e s s . I n t hi s s t udy , we i nc or por a t e d t he

“ WI DE” pl a t f or m ( We b-based Instructional Design Environment). WIDE was

developed by our research group in 2000 (Hou, 2000; Sung, Hou & Chang, 2001;

Chang, Sung, Hou, 2006), this study continually revised and added new modules

f or t e ac he r s ’ onl i ne i ns t r uc t i ona l de s i gn, c onduc t i ng bl e nde d l e ar ni ng a nd

knowledge sharing in an e-Learning context; the new version with knowledge

sharing f unc t i ons i s c a l l e d “ WI DE-KM” . Fi gur e 1 s hows t he homepa ge of

WIDE-KM, which displays major functions for teachers.

Figure 1 The homepage of WIDE-KM, which displays major functions for teachers.

WIDE provides an environment for teachers to design their own

e-instructional plan, conduct online teaching and various interactive learning

activities according to their designed e-ID plan, and fits with the execution

requirements of an online PBL teaching activity. For detailed introduction and

modules of the WIDE/WIDE-KM system, please kindly refer to Appendix A. The

following table lists WIDE functions related to online PBL of this research:

Table 2 Functions related to online PBL provided by the WIDE system

Function Description Upload teaching

materials

This function allows teachers to upload related resources gathered on the internet and their teaching plans and materials.

View teaching materials

This function allows students to download and read the teaching materials provided by the teacher.

Upload project reports

This function allows students to upload their report files.

Vi e w pee r s ’ reports

Thi s f unc t i on a l l ows s t ude nt s t o downl oa d a nd r e a d t he i r pee r s ’ reports.

Asynchronous forum

These functions allow students to conduct online asynchronous discussions. Three sub-functions are available:

1. List of topics: All topics posted will be listed for users to browse. The users can click the topic link to enter the page of a specific topic and read the content of the topic and all responses.

2. Post new topic: After the user enters the title and content, the topic will be added into a list of topics and arranged according by time.

3. Reply to the topic: The user clicks on this function to post a response on the page of a certain topic. After completing the response, the reply will be added to the topic page.

3.2.2.2 Coding Scheme

In order to understand the level (depth) and status of knowledge

construction in the discussion process, the Interaction Analysis Model (IAM)

coding scheme proposed by Gunawardena, Lowe, and Anderson (1997) was

chosen for this study. This coding scheme is divided into five phases (see Table 3);

each phase represents a type of knowledge construction in the discussion content.

Table 3 Gu nawar de na, Lowe , and Ande r s on’ s I nt er ac t i on Anal y s i s Mode l ( I AM)

Code Phase Operation

C1 Sharing / comparing of information

Statement of observation or opinion;

statement of agreement between participants

C2 Discovery and exploration of dissonance or inconsistency among participants

Identifying areas of disagreement;

asking and answering questions to clarify disagreement

C3 Negotiation of

meaning/co-construction of knowledge

Negotiating meanings of terms and negotiation of the relative weight to be used for various agreement

C4 Testing and modification of proposed synthesis or co-construction

Testing the proposed new knowledge against existing cognitive schema, personal experience or other sources C5 Agreement

statement(s)/application of newly constructed meaning

Summarizing agreement and

meta-cognitive statements that show new knowledge construction

C6 Others Discussions irrelevant to knowledge construction

This coding scheme is commonly used in the analysis of online discussions

(Marra, Moore & Klimczak, 2004; Jeong, 2003), which enhances the validity of

content analysis (Rourke & Anderson, 2004). Jeong (2003), for example, used

IAM to understand the content of online discussions formulated by students

without teacher interventions. Learning from the experience of Jeong (2003), this

study seeks to understand the depth and sequential relationship of knowledge

construction by using IAM to code each posted message.

3.2.3 Design

This study analyzes the content of the knowledge sharing discussion in both

quantitative and qualitative approaches. Three analysis methods are adopted:

quantitative content analysis, lag sequence analysis and original protocol analysis.

Qua nt i t a t i ve c ont e nt a na l ys i s e xpl or e s t he l e ve l / de pt h of l e a r ne r s ’ knowl e dge

construction, while sequential pattern of discussion behaviors of members of the

learner community are analyzed by lag sequence analysis. This study also extracted

some original discussion text, and processed it with original protocol analysis to

trace and discuss derived content/behavior patterns. Discussions, reviews and

comparisons are carried out with results obtained from qualitative interpretation,

quantitative content analysis, and sequential analysis.

3.2.4 Procedures

This study conducted a PBL activity, implemented from the end of March to the

middle of May in 2004. Using the above peer assessment online discussion

mechanism, the teacher in the activity requested each student to complete a project

report within a month. Each student needed to independently look for a real and

operating company, analyze its operational bottlenecks, and design an e-learning

training plan for the company employees. Each student was asked to upload the

complete training plan to the WIDE system and post an article explaining the

background of their chosen company and illustrating the e-Learning training

program for the company in their report. From May 9

to May 20

, 2004, all

students were required to conduct peer assessment discussion activities. In other

words, each student nee de d t o r e vi e w pe e r s ’ r e por t s a nd a s s e s s t he r e por t s by

commenting on the topics. The students were informed that this peer assessment

activity was a part of their semester grade and that the teacher would not participate

in any comments, replies, or guidance throughout the discussion activity.

3.2.5 Data Analysis

45 students participated in this study. With one project per student and one topic

per project, there were 45 topics in all. Each topic had messages (including the

introduction article, replies by others, and replies by the student who announced the

topic), with a total of 329 message records at the end of the activity.

The data was then coded by a rater of the research group based on the

above-mentioned IAM knowledge construction coding scheme. The coding method

used each topic as a unit and then coded each message based on its chronological

order. During the coding process, if two or more paragraphs belonged to different

codes within a message, the message would be assigned two or more codes based on

the order of the content (e.g., if the first section of a message belonged to the C1

category, and the second and third belonged to the C3 category, then the coding for

the message would be C1, C3). After all messages had been coded, each topic had a

set of coding data. Coding yielded 332 codes. To further examine the reliability of

the codes, we randomly selected 167 messages (about 50% of the messages) to be

analyzed by a second rater. The Kappa reliability of the codes was 0.727 (p<0.01).

The coded data then underwent sequential analysis and content analysis for

knowledge construction.

3.3 Results and Discussions

3.3.1 Content Analysis

The total numbers and percentages of each code are shown in Figure. 2:

Figure 2 Scale map of the coding of knowledge construction in the online

The distribution of coding shows that, in the process of student discussions,

knowledge construction was mostly composed of C1 (sharing/comparing of

information or proposing similar ideas) (69.88%), followed by C2 (discovery and

exploration of dissonance or inconsistency among participants) (8.73%). C3

(negotiation of meaning/co-construction of knowledge) (0.3%) was very rare. C4

and C5 were not observed.

This suggests that some aspects of knowledge construction, such as C3, C4,

and C5, are more difficult to reach, as proposed using the same coding scheme by

Gunawardena, Lowe, and Anderson (1997).

In order to observe the differences between ordinary online discussions and

those that involve peer assessment, we also compared our findings with those of

Jeong (2003), which focused on free discussions on topics specified by a teacher

who also did not intervene in the discussion process (C1: 93.7%, C2: 2.4%, C3:

1.9%, C4: 1.0%, C5: 1.9%). The higher percentage of C2 (discovery and exploration

of dissonance or inconsistency among participants) in our study shows that, since

s t ude nt s we r e r e qui r e d t o obs e r ve a nd c omme nt on e a c h ot he r ’ s wor k dur i ng t he

process of peer assessment, information was not only shared among students but

different perspectives were also proposed, examined, and discussed, allowing the

knowledge construction of this aspect. Free discussions that do not involve peer a s s e s s me nt , howe ve r , ar e of t e n l i mi t e d t o s ha r i ng a nd c ompa r i ng e a ch ot he r ’ s

information or proposing similar opinions.

The results show that in a peer-assessment knowledge sharing discussion

activity, learners may think deeply about a topic and propose different perspectives.

Thi s may pr omot e l e a r ner s ’ i nt e r na l i za t i on of pe e r s ’ wor ks or discussion content and

externalize their own perspectives.

This study also discovered that, in the context of peer assessment, the addition

of a peer assessment knowledge sharing discussion mechanism did not increase s t ude nt s ’ di s c us s i ons i n t he a s pe cts of C3 (negotiation of meaning/co-construction of

knowledge), C4 (testing proposed new knowledge against existing cognitive schema,

personal experience, or other sources), or C5 (summarizing and showing new

knowledge construction). This also shows that while the students could share their

projects, propose similar ideas (C1), and explore and discuss their different

perspectives (C2), they were unable to reach in-depth knowledge construction and

negotiate knowledge meaning (C3), test proposed new knowledge against existing

sources (C4), or summarize and generate new knowledge (C5). This is similar to the

findings of Jeong (2003) and Gunawardena, Lowe, and Anderson (1997). On the

other hand, we observed that there are 70 C6 codes in the discussions (irrelevant to

knowledge construction), forming 21.1% of the 332 total codes. This suggests that, a f t e r a s t ude nt ’ s pee r s ha ve ma de c omme nt s t ha t a gr e e or di s agr ee wi t h t he

pe r s pe c t i ve s i n t he s t udent ’ s r e por t , t he s t ude nt ’ s f ol l ow-up explanations or replies

do not seem to address the knowledge content and conduct knowledge

internalization/externalization at a deeper level. Moreover, these 21.1% of

discussions were irrelevant to knowledge construction, straying from the main

topics.

3.3.2 Sequential Analysis

Through the above analysis of quantities of each aspect of knowledge

construction, we could only understand the percentage of each aspect of knowledge

construction in the discussion. However, these percentages did not lead to a clear

understanding of the stude nt s ’ be ha vi or a l pa t t e r n i n t he overall discussion process

(i.e., the pattern of how the students finish a discussion based on a certain aspect of

knowledge construction and then shift to continue with another discussion based on

another aspect of knowledge construction). To address this issue, we used sequential

analysis that allowed us to statistically examine among which behaviors a significant

sequential correlation exists and construct a diagram about it. We were then able to

infer behavioral patterns in the discussion process, facilitating our diagnosis of

insufficiency in the discussions.

We compiled all the topics in the coding data: 45 topics with 329 messages.

After coding, there were a total of 332 codes. Since sequential analysis focuses on

how a certain behavior immediately follows another, we calculated the frequency of

each behavioral category immediately following another behavioral category (C4

and C5 were excluded because they did not occur). The results are listed in Table 3,

where each row represents the category of the initial code and each column is the

code of the behavior that follows immediately afterwards. The numbers in the table

represent the total number of times (total frequency) a column behavior follows

immediately after a row behavior in the total coded messages (for example, the numbe r “ 16” i n r ow 2/ c ol umn 1 me a ns t ha t “ C1 oc c ur r i ng i mme di a t e l y af t e r C2” 16

times). In order to infer whether each sequential correlation reached statistical

significance, we conducted a sequential analysis (Bakeman & Gottman, 1997) based

on the information listed in Table 4. The results were used to form the Adjusted

Residuals Table in Table 5, i n whi c h i f a s e que nc e’ s Z-value was greater than +1.96,

the connectivity of this sequence reached statistical significance (p<0.05). Based on

this table, we were then able to draw a behavioral transition diagram (see Figure 2).

Table 4 Frequency transition table

C1 C2 C3 C6

C1 140 20 1 32

C2 16 5 0 5

C3 0 0 0 1

C6 30 4 0 32

Table 5 Adjusted residuals table (Z-scores)

C1 C2 C3 C6

C1 3.98 0.30 1.22 -6.82

C2 -0.24 1.60 -0.33 -0.59

C3 -0.81 -0.32 -0.06 1.53

C6 -2.56 -1.35 -0.62 5.13

Figure 3 Behavioral transition diagram for peer assessment online discussions

Figure 3 shows the sequences in Table 5 that reached statistical significance.

The numerical values represent the Z-values for sequences, and the arrow heads

with different thicknesses represent different levels of significance and transitional

directions. From the information listed in Table 4 and Figure 3, we have a basic unde r s t a ndi ng of t he s e que nt i a l t e nde nc i e s of t he s t ude nt s ’ be ha vi or s . The pa t t e r ns

of behavioral transitions have been compiled and are listed below:

(1) During the discussion process, the only sequences that reached significance

were C1->C1 and C6->C6; the other sequences did not reach significance.

(2) The behaviors of C1 (sharing, comparing information, or proposing similar

ideas) and C6 (messages irrelevant to knowledge construction) show repeated

occurrence, both have no sequential correlations with C2 or C3.

For C1->C1, we can see that when the students share, compare, or propose

similar ideas, they also demonstrate a certain level of dedication or consistency in

their discussion behaviors, and may promote the consistency of knowledge

internalization/externalization at the C1 phase.

However, their discussion behaviors do not have a sequential correlation with

other discussions related to knowledge construction. In other words, while the

students shared or compared information or proposed similar ideas (C1), they were

unable to extend their discussions or propose or explore other perspectives (C2),

negotiate the knowledge meaning (C3), test proposed new knowledge against

existing sources (C4), or summarize and generate new knowledge (C5). Although in

the previous analysis, we already understood that during peer assessment knowledge s ha r i ng di s c us s i ons , t he pe r c e nt age of t he s t ude nt s ’ be ha vi or of pr opos i ng or

exploring different perspectives (C2) occupies the total behaviors to a certain extent

(8.73%), Figure 3 shows us that the behavior of C2 itself has not reached continuity

(C2->C2), and has not formed a significant sequential correlation with other

behaviors. On the other hand, C6->C6 shows that once the behavior of posting

irrelevant discussion messages has occurred, this behavior will form a level of

continuity that affects normal discussions. The analysis in the previous section

shows that irrelevant discussions take up 21% of total discussions. This may divert f oc us i n s t ude nt s ’ di s c us s i ons t o di r e c t i ons unr e l a t e d t o t he ma i n t opi c .

To s ome e xt e nt , t he s e f i ndi ngs a l l ow us t o under s t a nd t he l i mi t s of s t ude nt s ’

peer-assessment online discussions when the teachers do not intervene. By

comparing these findings to the analysis in the previous section, we have an initial

understanding that student discussions mostly consist of sharing or comparing

information or proposing similar ideas, followed by proposing or discussing

different opinions. However, whether students have similar or different opinions on

a certain project, the discussions fail to generate more aspects of knowledge

construction. On the other hand, the irrelevant discussions observed in this study

also tended to continue and thus shift the focus.

Insufficiency in the depth and width of these discussions is similar to what was

found in other studies regarding the implementation of online PBL. Some studies f ound t ha t l e a r ne r s of t e n di r e c t l y vi e w onl i ne r e s our c e s a s t he “ a ns we r s ” t o pr oj e cts

without in-depth cognitive processing or discussions (Wallace & Kupperman, 1997;

Chang & McDaniel, 1995), or may make inappropriate inferences regarding the

topics (Krajcik, et al., 1998). Through the analysis in this study, we can see that even

though peer assessment may provoke a certain level of behaviors related to

knowledge construction where peers propose comments or different opinions (C2:

8.73% ), it does not seem like the students were able to extend the depth and width

of their knowledge construction (e.g., generate significant sequences of C1->C2,

C1->C3, C1->C4, C1->C5, C2->C3, C2->C4, C2->C5 in Fig. 2), or conduct deeper

interactive discussions, such as proposing new understanding by comparing what is

being discussed with previous experience (C4->C5) or exploring and proposing

different opinions regarding the new understanding (C5->C2).

3.3.3 Original Protocol Analysis

In order to explore the above findings and possible causes for these

phenomena, we extracted reply messages for the topic of student S041 in the

discussion, shown in Table 6:

Table 6 Extraction of an actual discussion example (Student no.: S041) Message

No.

Author Content Time

#4101 S041 My topic introduces the teaching system of customer service of Company A.

4/24 07:50:21

#4102 S005 I found that your analysis is exactly the s a me a s S124~…s o…di d one of you upload the wrong data?

5/10 22:26:48

#4103 S041 Well~ if there is repetition, I will charge for copyright!

5/11 12:39:52

#4104 S007 The font in your report is too small and I suggest a larger font, otherwise its such an eyesore to read it!~~

…

5/11 14:42:15

#4105 S019 A big round of applause for you first, the content is very professional and the analysis quite detailed. I could not identify any fault, please keep the hard work goi ng! Go go go! ! …

5/11 23:15:48

#4106 S012 For this system, is it possible that students cannot effectively grasp the essence due to the extensive aspect of function, which causes bad learning efficiency?

5/15 16:51:41

#4107 S041 The system design adopts table and procedures often used in companies; many settings and items of development tools are omitted, so it will be faster during operation.

5/17 08:35:08

From the discussion in Table 6, the following can be seen: First, , the doubt of

student S005 about mistaken uploading of file, the suggestion from S007 about font

size and the superficial praises of S019 both did not address the knowledge of the

construction (e.g. reply from S041 to S005 in #4103) and induce C6->C6, which

may have affected the continuity of the entire knowledge construction.

As for the opinion of S012 and reply from S041 (#4106, #4107), while S012

provided an opinion, he did not describe it in a precise and practical way (e.g.:

provide description with actual example or supplement with sufficient evidence to

explain the doubt). This may have biased the direction of inference or lack of de t a i l e d e xpl a na t i on i n S041’ s r e pl y ( e . g. : i n t hi s c a s e , S041’ s r e pl y di d not

completely answer the question raised by S012. He did not describe what the

so-c a l l e d “ s e t t i ngs a nd i t e ms of de ve l opme nt t ool s ” a r e or e xpl a i n i f ope r a t i on

efficiency and learning efficiency directly relate to each other). It can be seen that if

there is an incidence of insufficient information or imprecise explanation in the

content of assessment discussion between students, deviation of explanation or

inference in the reply can be induced, which interferes with the achievement of more

phases of knowledge constructions (e.g.: C3, C4, C5).

3.4 Summarizations

In sub-study I, when the learner community was using peer assessment strategy,

their knowledge-construction shows a certain level of continuity at the C1 phase, and

the result of C2 (8.7%) was also higher than the findings in a study by Jeong (2003)

in which free discussions were based on topics specified by teachers. This suggests

that when using peer assessment, students may externalize different perspectives.

Regarding limitations, we found that 1/5 of all discussions were irrelevant and

were continued. In other words, the percentage of off-topic discussions was not just

high but tended to continue once such a discussion occurs. The depth of

knowledge-construction was mostly limited to the phases of C1 and C2, and could not

extend to other phases. When we further look at our original protocol analysis, we

find that the C6 code contained contents irrelevant to knowledge-discussion,

including superficial praise and comments on the formats of the files/documents.