Overview
To obtain a more complete picture of students’ self-reports toward their usage of cognitive operations in reading process, this study aims to investigate the cognitive operations on EFL university students’ reading comprehension by using both quantitative and qualitative methods in the research design, data collection, and data analysis. The detailed information of the research methodology is indicated as follows. The first section describes the subjects participating in this study. The second section shows sampling strategies for this study. The third section refers to the research tools utilized in the research.
The fourth section refers to data collection procedures. The last section indicates data analyses for this study. Each section is elucidated elaborately.
Subjects
Subjects in this study were 120 students composed of 22 seniors, 17 juniors, 53 sophomores, and 28 freshmen, majoring in Applied English at I-Shou University participating on a voluntary basis. The subjects encompassed 32 males and 88 females ranging from 19 to 25 years old, with a mean age of 21.5. A demographic questionnaire was provided to collect subjects’ background information. Results from the questionnaire revealed that all of the subjects had received formal English instruction for an average of 9 years. To evaluate students’ reading proficiency levels, a reading
comprehension test composing of three types of test items (see Appendix A) was conducted on the spring semester of the 2010 academic year in order to categorize those subjects into three groups — low-, intermediate-, and high-English proficiency levels (EPL).
There were 10 items in each type of the reading test including identifying main ideas, locating details, and making inferences. Subjects would obtain one point if they answered correctly in the multiple-choice test.
The test results ranged from 3 to 29 points, with the mean score of 12.5 and the median of 12. Thirty-two subjects (top 27%) whose scores ranged from 16 to 29 were labeled “high,” 46 subjects (46%) with the scores of 11 to 15 were labeled “intermediate,” and 42 subjects (bottom 27%) with the scores of 3 to 10 were labeled “low.” There was a significant difference between the high and low groups, indicating that the high group scored significantly higher (M
= 18.72, SD = 3.06) than the low group (M = 7.88, SD = 2.00). Thus, the number of subjects in each group was quite appropriate. The detailed information regarding the distribution of subjects is shown in Table 3.
Table 3
Classification of Three Groups
Level N Rate Score M SD
High-EPL 32 27% 16 - 29 18.72 3.06
Intermediate-EPL 46 46% 11 - 15 12.39 1.13
Low-EPL 42 27% 3 - 10 7.88 2.00
Note. N=120
Sampling Strategies
To capture an adequate sample size from a larger population pool, both quantitative and qualitative sampling strategies composing of multistage sampling and stratified purposeful sampling, were employed in the present study for obtaining the significant research results and reaching the generalizability.
Quantitative Sampling
Multistage sampling, one of the probability sampling strategies, including cluster sampling and random sampling was utilized to select the subjects because it is more reliable even though it needs more complex calculations. The cluster sampling was employed first to select students ranging from freshmen to seniors, majoring in Applied English at I-Shou University. Subsequently, random sampling was applied to select a sample with 120 students among these four grade levels. Thus, the sample size, 120 students majoring in English at I-Shou University, was chosen to do the questionnaire survey for attaining their self-reports toward their cognitive operations.
Qualitative Sampling
In addition, stratified purposeful sampling was applied in the qualitative interview method because those 15 participants, five from each group, randomly chosen from the 120 students needed to have the experiences of using their cognitive operations in terms of identifying main ideas, locating
details, and making inferences while reading. In this research, the number of the 15 participants was decided based upon the adequacy and saturation of their information; in other words, the information collected from those 15 participants, stratified by their three proficiency levels, was sufficient to analyze the research results. Therefore, 15 out of 120 students were selected to explore their in-depth perceptions toward their cognitive operations in the reading process.
Summary
Both quantitative and qualitative sampling strategies composing of multistage sampling (cluster and random sampling) and stratified purposeful sampling were employed in the present study, in order to make sure that the participants were sampled appropriately and sufficiently enhance the representativeness of research results.
Instrumentation
In order to investigate the relationship between English proficiency levels and students’ cognitive operations in reading, three types of instruments were used in the study: (1) a reading comprehension test, (2) a questionnaire, and (3) a semi-structured interview technique to canvass deeper perceptions toward the conditions of students’ cognitive operations while doing the three types of reading test. More detailed information about the instruments is expatiated as follows.
A Reading Comprehension Test
A reading comprehension test consisting of the items of identifying main ideas, locating details, and making inferences, was extracted from Broukal’s (1995) and Phillips’s (1996) books pertinent to TOEFL reading comprehension test (see Appendix A). “TOEFL” represents Test of English as a Foreign Language, which is an international test to scrutinize non-native English students’ English ability (Peirce, 1992). The major reason to select the three types of test items from Broukal’s and Phillips’s books is that they divided the simulated TOEFL reading test into three sections — main ideas, details, and inferences — which fit in with the aim of the present study.
There were 10 items in each type of the reading test and the duration of the test was 50 minutes in total to assure that the subjects had enough time to finish all of the test items. The number of words in each passage was approximately 200 words. A pilot testing was conducted by the first three participants in order to ensure the content to be comprehensible without the interference of cultural knowledge. From the pilot testers’ suggestions, only some vague words were modified for better understanding.
A Questionnaire Design
A questionnaire, a nonexperimental research, was designed in the present study to investigate students’ self-reports of their cognitive operations while reading. The questionnaire included three sections of the reading cognitive operations in terms of identifying main ideas, locating details, and
making inferences. Thirty questions in this questionnaire were composed of 10 questions in the main-idea section, 11 questions in the detailed section, and 9 questions in the inferential section (see Appendix B). What is more, a five-point Likert scale ranging from “5” (“Strongly agree”) to “1” (“Strongly disagree”) was administered to examine students’ perceptions of cognitive operations. In order to testify validity and reliability of the questionnaire, each questionnaire item was not only adopted from several researchers’ studies (Lai, 2006; Shih, 2008; Yeh, 2006), but also examined by using the software, SPSS 17.0, showing that there was a significant difference in each questionnaire item, so all the items should be retained. As to the result of reliability, the Cronbach Internal Consistency Coefficients for the types of main idea, detail, and inference were .79, .76, and .78, respectively. The detailed information is shown in Table 4.
Table 4
Results of Cronbach Internal Consistency Coefficients
Type N of Items Cronbach’s Alpha
Main idea 10 .79
Detail 11 .76
Inference 9 .78
Note. N=30
Semi-structured Interview
Fifteen participants were interviewed individually by using a semi-structured interview technique because it is essential to gather students’
in-depth perceptions toward their usage of cognitive operations while reading.
Stratified purposeful sampling was applied in this study. Five participants from each proficiency level were randomly selected for doing the interview.
In order to protect the participants from harm, they were informed by a consent form (see Appendix C), indicating that they had not only the right to refuse to answer the sensitive questions, but also the volition to withdraw from the interview process any time. Moreover, participants were notified that their personal identities would not be uncovered because their names were replaced with numbers. Each participant was interviewed about 15 to 20 minutes. During the interview process, each participant might answer the questions in either English or Chinese depending on what language the participants could express their own thoughts explicitly. Furthermore, more open-ended answers related to the topic of the present study were encouraged and explored. Sixteen interview questions (see Appendix D) primarily focused on exploring the participants’ past, present experiences, and perceptions toward their use of cognitive operations in doing the comprehension test. A pilot testing was done by the first two participants to ensure the validity of the interview questions. Some adjustments were made including the revision of the appropriate tense, the modification of some questions’ distortion, and the clarification of the ambiguous questions.
Data Collection Procedure
The pragmatic parallel mixed-method was mainly applied in this study;
that is, quantitative and qualitative data were collected simultaneously or with a small time lag. During the process of quantitative data collection, a questionnaire and a reading comprehension test composing of main-idea, detailed, and inferential test items were conducted. In the beginning of the semester, a reading comprehension tests taking 50 minutes was used to investigate students’ English proficiency levels and classify them into three groups depending on the outcomes of their reading performance.
Subsequently, a questionnaire was conducted immediately after the reading test, in order to explore students’ self-reports toward cognitive operation usage in terms of identifying main ideas, locating details, and making inferences while reading.
Finally, after the process of quantitative data collection, individual interview was performed to obtain more in-depth information about the students’ perceptions and feedback toward the condition of their cognitive operations. Five students from each group were selected randomly to participate in the interview. Tape recording was used to collect participants’
perceptions; moreover, participants’ answers were transcribed by the researcher. The design of the research process is illustrated in Figure 1.
A Reading Comprehension Test High
Interview
Data Collection
Data Analysis
Identifying Main Ideas Locating
Details Making Inferences Inter-
mediate Low
Questionnaire
Figure 1. The Process of Research Design
Data Analysis
In this study, both quantitative and qualitative research techniques were utilized to explore the following three research questions: (1) Is there a significant difference among students with different proficiency levels on their usage of cognitive operations in terms of identifying main ideas, locating details, and making inferences? (2) What is the relationship between students’
English proficiency levels and their usage of cognitive operations? (3) What are students’ perceptions toward their application of cognitive operations in reading comprehension?
Proficiency Levels In order to scrutinize the difference of students’ English proficiency levels on their usage of cognitive reading operation, a one-way analysis of variance (ANOVA), which the independent variable is the students’ English proficiency levels and the dependent variable is their self-reports of the three cognitive operations, was implemented to investigate whether students with different English proficiency levels had significant differences on their usage of cognitive operations. The diagram by employing an ANOVA is shown in Figure 2.
Figure 2. The Diagram between English Proficiency Levels and Self-Reports of Cognitive Operations
Additionally, a bivariate (zero-order) correlation analysis was employed to investigate the relationship between the variables of English proficiency levels and students’ self-reports of their usage of cognitive operations. More specifically, a correlation coefficient (r) was calculated to examine whether
these two variables were significantly correlated. Another consideration with respect to the significance of correlations is the magnitude (strength) and direction of the correlations. The number that represents the correlation can range from -1.00 to +1.00. A high positive value represents a high positive relationship, a low positive value represents a low positive relationship, and so on (McMillan & Schumacher, 2006).
Finally, a semi-structured interview technique was applied to explore more in-depth information about students’ perceptions toward their usage of cognitive operations. Tape recording was employed during the interview process. After the interview, all of the participants’ expressions were transcribed by the interviewer in order to analyze participants’ perceptions regarding their cognitive operations while doing the reading comprehension test. After transcribing, the interesting units were marked and labelled for further content analysis.
Summary
Regarding the research design of the present study, it is necessary to clarify and understand an elaborate methodology, in order to ensure the practicability of the whole research. First of all, subjects in this study were 120 students majoring in English at I-Shou University. Both quantitative and qualitative sampling strategies composing of multistage and stratified purposeful sampling were employed in the present study to select the participants from a larger population pool. In addition, a TOEFL reading
comprehension test was used in order to categorize the subjects into three proficiency levels — low, intermediate, and high levels. In addition, the subjects were required to do a questionnaire immediately after the reading test, in order to investigate their usage of cognitive operations in terms of identifying main ideas, locating details, and generating inferences while reading. The pilot testing of the reading comprehension test and the questionnaire was employed in advance to make the content more comprehensible without the interference of cultural knowledge and vague words. What is more, 15 participants, stratified by their three proficiency levels, were selected randomly from the 120 students to participate in the interview, in order to capture their in-depth perceptions toward their usage of cognitive operations in reading process. After data collection, a one-way analysis of variance (ANOVA), a correlation analysis, and a semi-structured interview technique were employed to investigate the difference and relationship between students’ English proficiency levels and their usage of cognitive operations.