CHAPTER 3: METHODOLOGY
3.1 Conceptual Factors
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
25
CHAPTER 3 METHODOLOGY
This study focuses on a review of KB studies from a worldwide perspective. Following the research questions stated in the introductory chapter, all KB studies articles from the Scopus database were processed by following these steps: (a) selection of KB studies using both inclusive and exclusive criteria (see below for details), (b) coding of related information for analysis, (c) analysis of the contribution patterns trends in KB studies.
3.1 Conceptual Factors
Five conceptual factors were examined in this study, including demographic factors, methodologies, online learning platforms, usage of KBPs, and learning outcomes in KB studies.
3.2 Selection of Knowledge Building (KB) Studies
Figure 3-1 illustrates the four steps used for selecting KB studies articles for the
systematic review. First, the Scopus database was selected as the favored searching engine in this study. The main reason was that the Scopus database is identified as a comprehensive database compared to Web of Science and Google Scholar. All results from the Web of Science and Google Scholar can be found in the Scopus database. Hence, the Scopus database was selected for searching articles related to studies of KB pedagogy. The Scopus database was searched with particular keywords and publication types for the initial search periods, which fell on November 23, 2016. This present study mainly focused on articles that were searched with the keyword of knowledge building (KB), as showed in Figure 3-2. The search resulted in 843 articles with the above-noted keyword appearing in the title, abstract, or keywords.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 3-1 Search Strategies Procedure
The next step was to filter irrelevant articles manually. By reviewing the abstracts of these articles, 529 articles were excluded since they defined knowledge building (KB) using a different definition from the one used in this study. The filtered process resulted in 314
articles which were further assessed using the following steps and which were required to meet some inclusive and exclusive criteria for further review. The searched paper/study was included if (a) it was an article written between 2001 and 2016, (b) it was an empirical study, (c) it involved some pedagogical design in KB, and (d) it involved the use of an online learning platform. However, the selected study was excluded if (a) it was a review paper, an unpublished dissertation, or a survey, or (b) it involved gifted students or students with learning disabilities as the participants. Eventually, 56 KB studies (see Appendix A) were selected for this review.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
27
Figure 3-2 Screenshot of Scopus Database Keyword Searching
A total of 22 journals were listed among these articles: (a) Asia Pacific Education Review, (b) Asia-Pacific Education Researcher, (c) Australasian Journal of Educational Technology, (d) British Journal of Educational Technology, (e) Canadian Journal of Science, Mathematics and Technology Education, (f) Computers and Education, (g) Early Education and Development, (h) Educational Technology and Society, (i) Educational Technology Research and Development, (j) IFIP International Federation for Information Processing, (k) Informatics in Education, (l) Instructional Science, (m) International Journal of
Computer-Supported Collaborative Learning, (n) International Journal of Continuing
Engineering Education and Life-Long Learning, (o) International Journal of Technology and Design Education, (p) Journal of Educational Computing Research, (q) Journal of
Mathematics Teacher Education, (r) Journal of Science Education and Technology, (s) Journal of the Learning Sciences, (t) Learning Environments Research, (u) Teachers College Record, (v) Turkish Online Journal of Educational Technology.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
After identifying the selected articles, a coding system was designed to perform further analysis. The coding details are provided in the following section.
3.3 Coding System
3.3.1 Bibliographic Coding
Two bibliographic categories were identified in coding the 56 selected articles including (a) Study ID number, (b) Publication year. These bibliographic categories were derived from coding manual examples of meta-analysis (Lipsey & Wilson, 2001). The Study ID number was coded from the initial search that yielded 843 results. Hence, the selected article ID numbers were between ID1 and ID843. There were 17 articles that included the letters a, b, or c behind the Study ID number with the conditions that these articles contain (a) at least two studies with participants of different education levels, or (b) at least two progressive studies with participants of the same education level. The publication year was directly coded with four digits, for instance 2016.
3.3.2 Demographic Coding
Table 3-1 shows the demographic coding included in this study. There were four items that included the countries where KB studies were conducted, education levels of participants, identities of participants, and subject domains researched in the selected KB studies. Twelve countries were identified in this study: (a) Australia, (b) Canada, (c) Finland, (d) Hong Kong, (e) Italy, (f) Japan, (g) Mainland China, (h) Mexico, (i) Singapore, (j) Taiwan, (k) Turkey, and (l) United States. Among these countries, two categories were identified in the coding: (a) Eastern countries, (b) Western countries. During the coding of different countries, for this study we employed two important criteria for categories. One was related to regions and another one was related to culture. Among these countries, Turkey was found to be difficult
‧
to categorize as either an Eastern country or a Western country, due to its complexity. From the different sources identified in Table 3-2, this study determined that Turkey was grouped in Eastern country based on culture perspective. Turkey was majority of Muslim and steeped in Middle Eastern traditions, which was closer to Eastern country (Mirabella, 2016).
Education level categories were derived and modified from the International Standard Classification of Education (ISCED, 2011). Eight categories of education levels were identified in the coding of this study: (a) Preschool, (b) Grade 1-6, (c) Grade 7-9, (d) Grade 10-12, (e) Undergraduate, (f) Graduate, (g) Postgraduate, (h) Not elsewhere classified.
Three categories of identities of participants were identified in the coding: (a)
Pre-service teacher, (b) In-service teacher, or (c) Learner. All of the participants’ categories were derived from the selected articles.
Twelve categories of subject domains were identified in the coding: (a) Chinese, (b) Design, (c) English, (d) Geography, (e) History, (f) Language, (g) Mathematics, (h) Pedagogy, (i) Science, (j) Technology, (k) Visual Art, or (l) Not specified. All the subject domain
categories were derived from the selected articles.
Table 3-1
Demographic Coding
Item Category
Country 1 Eastern country 2 Western country
Education level 1 Preschool 2 Grade 1-6 3 Grade 7-9 4 Grade 10-12 5 Undergraduate 6 Graduate 7 Postgraduate 9 Not elsewhere
classified Identities of
participants
1 Pre-service teacher 2 In-service teacher 3 Learner
Subject domains 1 Chinese 2 Design 3 English
4 Geography 5 History 6 Language
7 Mathematics 8 Pedagogy 9 Science 10 Technology 11 Visual Art 12 Not specified
‧
Turkey as Eastern Country or Western Country Supported by Different Sources Turkey is marked as country of
Eastern Western Remarked
The IUCN Red List partly in Asia and Europe
Worldatlas (n.d.) X Asia
Mirabella (2016) X X Geographic of
Turkey was closer to Europe, but culture
3.3.3 Research Design Coding
Table 3-3 shows the research design coding included in this study. There were two items that included research designs and analysis methods among these selected KB studies. A total of seven categories of research designs were identified in the coding of this study, which include (a) Case study, (b) Design experiment, (c) Ethnography, (d) True experiment design,
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
31
(e) Quasi-experiment design, (f) Exploratory study, and (g) Longitudinal study. All the research design categories were derived from the selected articles. Hence, each selected article research design was coded from the description of methodology section. However, there were 22 articles that did not clarify the research design employed. To solve the coding problem on these research designs, some solutions were proposed to be discussed with the inter-coder through face-to-face contact.
In addition, three categories of analysis methods approaches were identified in the coding: (a) Quantitative analysis, (b) Qualitative analysis, and (c) Mixed methods analysis.
These categories of analysis method approaches were derived from the selected articles.
Quantitative analysis was conducted by adopting statistical methods in analyzing data, for example pre- and post-tests, correlation analysis, analysis of variance (ANOVA), and so forth.
Qualitative analysis was conducted by analyzing data from interviews, classroom
observations, written productions or sketches, and so forth. Mixed methods analysis was conducted by applying both quantitative analysis methods and qualitative analysis methods in analyzing data. For instance, ID27b involved adopting qualitative content analysis in
analyzing content of discussion of learners and, ID27b also used a process of adopting t-tests for comparing two groups from the level of participation. Qualitative findings supported by quantitative results were found in most of the KB studies that adopted mixed methods
analysis. During the coding process of analysis methods, each article was validated according to the description of the data analysis section.
‧
Research designs 1 Case study 2 Design experiment 3 Ethnography 4 Exploratory study 5 Longitudinal study
6 Quasi-experiment design
7 True experiment design
Analysis methods 1 Quantitative analysis
2 Qualitative analysis
3 Mixed methods analysis
3.3.4 Online Learning Platform Coding
Table 3-4 shows the online learning platform coding included in this study. Two
categories of online learning platforms were identified in the coding: (a) KF and (b) Non-KF.
All the online learning platform categories were derived from the selected articles. During the coding process on online learning platforms, each article was validated according to the description of the online learning environment.
Table 3-4
Online Learning Platforms Coding
Item Category
Online learning platforms 1 KF 2 Non-KF
3.3.5 Knowledge Building Principles (KBPs) Coding
Table 3-5 shows the KBPs coding included in this study. The KBP categories were derived from Scardamalia and Bereiter (2010). Twelve categories of KBPs were identified in the coding: (a) Real ideas, authentic problems, (b) Improvable ideas, (c) Idea diversity, (d) Rise above, (e) Epistemic agency, (f) Community knowledge, collective responsibility, (g) Democratizing knowledge, (h) Symmetric knowledge advancement, (i) Pervasive KB, (j) Constructive uses of authoritative sources, (k) KB discourse, (l) Concurrent, embedded, and transformative assessment. During the coding process on KBPs, each article was validated
‧
according to the description in the following condition: (a) Literature review section, (b) Online learning platforms and KB tools, and (c) Pedagogical designs. All the ambiguous coding problems were solved via inter-coder agreement through face-to-face discussions.
Table 3-5 KBPs Coding
Item Category
KBPs 1 Real ideas, authentic problems
2 Improvable ideas 3 Idea diversity
4 Rise above 5 Epistemic agency 6 Community knowledge, collective responsibility
10 Constructive uses of authoritative sources
11 KB discourse 12 Concurrent, embedded and transformative assessment
3.3.6 Learning Outcomes Coding
The learning outcomes categories were derived from the selected articles and compared with the 21st century skills competencies (National Research Council, 2012).The National Research Council is an authoritative non-profit institution that publishes valuable articles. For this reason, this study chose the data source above as a main reference for comparing learning outcomes categories for this study. Three categories of learning outcomes were identified in the coding: (a) Cognitive domain, (b) Metacognitive domain, and (c) Social domain. Each category consisted of two to three subcategories which were discussed as coding categories with an inter-coder. These subcategories consist of basic and higher-order learning outcomes.
Basic cognitive learning outcomes include basic literacy and domain-specific literacy, while epistemic literacy was marked as a higher-order cognitive learning outcome. Basic
metacognitive learning outcomes involved reflection, while self-regulation and problem
‧
solving were parts of higher-order, metacognitive learning outcomes. A basic social learning outcome included interaction or participation, while metadiscourse was marked as a
higher-order social learning outcome. Table 3-6 illustrates the descriptions of the subcategories of learning outcomes coded in this study.
Table 3-6
Subcategories Learning Outcomes
Category Subcategory Description
Cognitive learning outcome
Basic Basic literacy Reading and writing competencies which identify the learning of new vocabularies.
Domain specific literacy
Subject domains competencies,
especially in Science and Mathematics.
Higher-order Epistemic literacy
Design thinking competencies which have a depth understanding of how knowledge is created.
Metacognitive learning outcome
Basic Reflection Ability to become aware through feedback from others as well as one’s self.
Higher-order Self-regulation Ability to sustain self-directed learning.
Problem solving Ability to use metacognition questions and statements to solve authentic problems.
Building on notes and reading others’
notes in KF. This category is
considered as basic interaction during the KB process.
Higher-order Metadiscourse KB discourse to sustain improvable ideas. This category focuses on higher-order interaction with others through building on rise above notes.
3.4 Validity
To conduct a systematic review on pedagogical design and implementation based on
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
35
KBPs from the Scopus database between 2001 and 2016, all coding categories were derived from the selected studies; therefore, these coding categories have high validity.
3.5 Inter-coder Reliability
After generating outcomes coding categories above, two coders worked together for learning outcomes coding. Coder A had expertise in implementing pedagogical design based on KBPs for 10 years. Coder B was a novice teacher who intended to adopt KB for History subject teaching. Both coders were required to categorize the key terms or sentences related to learning outcomes. A total of 113 key terms or sentences (referred to Appendix B) were to individually categorized by both coders into the learning outcomes and coded as outlined above. After they finished coding, both coders were required to discuss the coding results one by one. During the discussion, they found discrepancies in some of the coding. The coders adopted a negotiated agreement approach for assessing inter-coder reliability in this study (Campbell, Quincy, Osserman, & Pedersen, 2013). By discussing the disagreements, the coders were able to reconcile and solve the discrepancies on learning outcomes coding.
The next process involved calculating the reliability and agreement on learning outcomes coding. This study employed Cohen’s kappa as the measure for inter-coder reliability. Figure 3-3 displays the formula of Cohen’s kappa coefficient. Andale (2017) highlighted that po was the relative observed agreement among coders, while pe was the hypothetical probability of chance agreement.
Figure 3-3 Cohen’s Kappa Coefficient Formula
‧
After calculating Cohen’s kappa coefficient through the five steps (referred to in Appendix B) provided by Andale, Cohen’s kappa coefficient needed to follow the
interpretation of Cohen’s kappa in Table 3-7. Eventually, Cohen’s kappa coefficient k = 0.8 indicated substantial agreement among the coders.
Table 3-7
Interpretation of Cohen’s Kappa
Kappa Agreement
< 0 Less than chance agreement 0.01 – 0.20 Slight agreement
0.21 – 0.40 Fair agreement 0.41 – 0.60 Moderate agreement 0.61 – 0.80 Substantial agreement 0.81 – 0.99 Almost perfect agreement
Note. Adapted from “Understanding interobserver agreement: The kappa statistic,” by A. J.
Viera and J. M. Garrett, 2005, Family Medicine, 37, p. 362.
3.6 Data Analysis
Table 3-8 illustrates the analysis used according to the research questions. All the coding was handled in Microsoft Excel. Overall, the data analysis in this study adopted a descriptive analysis to reveal findings in the following chapter.
Table 3-8
Data Analysis with RQs
RQ Data Analysis
RQ1: What is the overall trend of KB studies over the past 16 years? Descriptive analysis RQ2: What common methodology is employed in KB studies? Descriptive analysis RQ3: What kind of online learning platform is commonly employed
in KB studies?
Descriptive analysis
RQ4: What KBPs are commonly used in KB studies? Descriptive analysis RQ5: What learning outcomes are commonly examined in KB
studies?
Descriptive analysis
‧
This chapter presents the findings collected from 56 selected KB studies that met the selection criteria mentioned earlier in this study. Descriptive analysis was adopted to illustrate findings with research questions respectively.
4.1 Current Trend of KB Studies
4.1.1 Patterns of Countries Where KB Studies Were Conducted
Table 4-1 shows that all 12 countries that conducted KB studies maintained active contribution to the journals over the past 16 years. Canada, Hong Kong, Singapore, and Taiwan were among the top four countries, while Italy, Japan, Australia, Finland, Turkey, Mainland China, Mexico, and the United States showed an emergent interest in KB pedagogy.
Based on the data collected, study ID559a was conducted with a cohort of graduate students from Canada and Hong Kong in 2007. Hence, the total of KB studies countries equaled n =57.
Among these countries, 53% were Eastern countries (n = 30) and 47% were Western countries (n = 27), which are shown in Figure 4-1.
Table 4-1
Ranking of Countries and Distribution Frequencies in Three Time Periods Time period
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 4-1 KB Studies Countries
Figure 4-2 shows that there was an upward trend for both Eastern countries and Western countries in the number of KB studies conducted over the three time periods. However, the number of KB studies conducted in Eastern countries is higher than Western countries between 2006 and 2016.
Figure 4-2 Trend of KB Studies Between Eastern and Western Countries in Three Time Periods
To explore the findings above, this study analyzed the trend of KB studies among 12 countries over three time periods. Figure 4-3 shows that with the exception of Australia,
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
39
Canada, Finland, Hong Kong, Japan, Singapore, Turkey, and United States, all the other countries showed an upward trend in the number of KB studies between 2001 and 2016.
Among these countries, Italy showed a transition from 0 to 5 in number of KB studies after the year 2011. Mainland China and Mexico also showed a transition from 0 to 1 in number of KB studies after 2011. Taiwan presented a rapid growth from 1 to 6 in the number of KB studies after 2011.
On the other hand, Figure 4-3 shows a surprising finding that Canada (n = 10) has the highest number of KB studies among all countries between 2006 and 2010. Although Canada shows a downward trend in the number of KB studies after the year 2011, the country that conducted the most KB studies was Canada (n = 16) over the three time periods. Another finding revealed that Australia, Hong Kong, Singapore, and Turkey had sustained the number of KB studies between 2006 and 2016.
Figure 4-3 Trend of KB Studies Among Countries Over Three Time Periods
4.1.2 Education Levels of KB Studies Participants
As the data shows, study ID339 was conducted using Grade 7-9 and Grade 10-12 participants. Hence, the total participants of KB studies equaled n = 57. Table 4-2 illustrates
‧
that 36.84% of KB studies engaged Grade 1-6 participants (n = 21), followed by 28.07%
involving undergraduate (n = 16) and 10.53% involving Grade 10-12 participants (n = 6). In contrast to the three educational levels stated above, 3.51% of KB studies engaged preschool (n = 2), Grade 7-9 (n = 2), graduate (n = 2) and postgraduate participants (n = 2) respectively.
Table 4-2
Percentage of Education Levels of Participants and Distribution Frequencies in Three Time Periods
Time period
Education level 2001-
2005
To explore the above findings, we had analyzed the trend of KB studies among eight educational levels over three time periods. Figure 4-4 shows that the number of KB studies that engaged Grade 7-9 and undergraduate participants showed an upward trend between 2006 and 2016. Grade 7-9 participants showed a transition from 0 to 2 in number of KB studies after the year 2011. Undergraduate participants presented a rapid growth from 3 to 12 in the number of KB studies after 2011.
On the other hand, Figure 4-4 shows a finding that the number of KB studies that
engaged Grade 1-6 participants (n = 13) was the highest between 2006 and 2010. Despite that the number of KB studies that engaged Grade 1-6 participants showed a downward trend after the year 2011, the most KB studies involved engaging in Grade 1-6 participants (n = 21) over three time periods. Another finding revealed that number of KB studies that involved preschool, Grade 10-12, and postgraduate participants was sustained between 2006 and 2016.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
41
Figure 4-4 Trend of KB Studies Among Educational Levels Over Three Time Periods
Figure 4-5 shows that most KB studies conducted in Eastern countries engaged Grade 10-12 and undergraduates as participants. In contrast, most of the KB studies conducted in Western countries engaged Grade 1-6 students as participants.
Figure 4-5 Comparison of Eastern Countries and Western Countries in Educational Levels
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
To look into participants’ identities in KB studies, Figure 4-6 shows that 67% were learners (n =37) and 33% were teachers (n =18). To explore the finding above, we analyzed the trend of KB studies among participants’ identities over three time periods. Figure 4-7 showed an upward trend for both learners and teachers over the three time periods. However, between 2006 and 2016 the number of KB studies that engaged learners was higher than those with teachers.
Figure 4-6 Participants’ Identities Engaged in KB Studies
Figure 4-7 Trend of KB Studies Among Participants’ Identities Over Three Time Periods
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
43
Table 4-3 illustrates that 78% of KB studies in Eastern countries engaged teachers (n = 14), while only 22% of KB studies in Western countries engaged teachers (n = 4). To explore this finding, this study analyzed the trend of KB studies that engaged teachers from Eastern countries and from Western countries over three time periods. Figure 4-8 shows that the number of KB studies that engaged both teachers from Eastern countries and teachers from Western countries was sustained between 2006 and 2016. However, the number of KB
studies that engaged teachers in Eastern countries was higher than those involving teachers in
studies that engaged teachers in Eastern countries was higher than those involving teachers in