Chapter 5 DATA AND MEASUREMENTS
An experiment of within subject analysis was conducted to test the hypotheses. A lab study with eye-tracking analysis was conducted to collect data, including TCAV system logs, questionnaire, and eye-tracking system logs.
We conducted a lab study to collect data from students who have taken at least one Java programming course. In total, data of 34 participants were analyzed, 17 of them (50%) were male, the average age was 21.03 (std = 1.1), 30 of them were from IS background, others included finance and sociology. 26 of them had experience of programming for more than 1 year. Subjective learning perception data were also collected through questionnaire. And both system log and eye-tracking data were collected for objective user behavior analysis.
Our research question could be answered by analyzing the data from user system log and questionnaire. But we also wanted to explore more on user’s behavior on the visualization analytics. As a result, objective eye-tracking data were applied to support our hypothesizes.
5-1 User Behavior and Perception Data
To measure the learning goal orientation, we used the 8-items measurement proposed by Zajac, Button, & Mathieu (1996). A sample item reads, “The opportunity to do challenging work is important to me.” Response were made on a 7-point scale (1=Strongly Disagree, 7 = Strongly Agree). Higher scores indicate higher learning goal orientation. One factor analysis was conducted to test the internal validity among the items. Item quality and factor correlation were satisfied and no item was needed to be dropped. The Cronbach’s alpha is 0.9, which support the internal consistency reliability of the factor. Appendix B provides the complete list of these items. The format of the graph visualization and the type of the learning task were recorded as participants
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
finishing each question in the lab study. To measure the learning comprehension, we calculated the average answer rate of search fact questions or inference generation questions for each user. For search fact questions, it was the rate that the participants checked the correct topics which met the query of the question. For inference generation questions, the same calculation was applied on Q1. For Q2 and Q3 of inference generation questions, we took them as a set questions and used the normalized discounted cumulative gain (NDCG) to measure the ranking quality. To measure the understanding of visualization, we asked the participants to answer what is the information the prior visualization mainly imply after finishing each iteration. The question list multiple choice of the possible information that the proposed visualization could imply. To calculate the correct answer rate, we set default answer for each visualization format (table 5.3). Finally the measurement of perceived learning is the 3-items measurement with 5-point scale (1 = Strongly Disagree, 5 = Strongly Agree).
The participants need to reply their perceived learning of the prior visualization analytics after finishing each iteration in the lab experiment. One factor analysis was conducted to test the internal validity among the items. Item quality and factor correlation were satisfied and no item was needed to be dropped. The Cronbach’s alpha is 0.92, which support the internal consistency reliability of the factor. The description of variables are summarized in table 5.1 and 5.2.
‧
Table 5.1 Description of user perceptions variables Research
variables Measurement items and description Learning
Comprehension Average correct answer rate that the participants checked the correct topics for search fact questions 1-3 and inference generation question 1. NDCG score for inference generation question 2-3.
Understanding of
Visualization Average correct answer rate that the participants reply of the following multiple selection question.
What is the information mainly imply in the prior visualization analytics (you may choose more than one response)?
Chapter 1 Show the trend of the answer rate.
Chapter 2 Compare the difference between individual and the class average
Chapter 3 Emphasize the extreme value belongs to specific topic of the individual or class average
Chapter 4 Show the Java topics involved in the question Chapter 5 Show the correlations of difficulty between the
Java topics.
Perceived
Learning The average score of the following 3 measurement items with 5-point scale (strongly disagree to strongly agree):
1. I learned knowledge of Java programming from the visualization analytics.
2. The visualization analytics helped me learn Java programming.
3. The visualization analytics improved my familiarity with Java programming.
‧
Table 5.2 Description of learning goal orientation, format and task variables Research
variables Measurement items and description Learning Goal
Orientation The average score of the following 8 items measurement with 7-point scale(1=Strongly Disagree, 7 = Strongly Agree). (Zajac et al., 1996)
A sample item reads, “The opportunity to do challenging work is important to me.”
Format Recorded as participants finishing each question in the lab study The value is either Radar, Bar or Line
Task Recorded as participants finishing each question in the lab study The value is either SearchFact or InferenceGeneration
Table 5.3 Default answer of understanding of visualization for each format Format type Information mainly imply
Line graphs 1. Show the trend of the answer rate.
3. Emphasize the extreme value belongs to specific topic of the individual or class average
4. Show the Java topics involved in the question
Bar graphs 2. Compare the difference between individual and the class average
4. Show the Java topics involved in the question
Radar graphs 2. Compare the difference between individual and the class average
3. Emphasize the extreme value belongs to specific topic of the individual or class average
4. Show the Java topics involved in the question
5. Show the correlations of difficulty between the Java topics.
‧
To answer our research question, regression analysis was adopted to explore the dependency between learning goal orientation, format, task and user perceptions. We integrated the collected data of these variables. In total, we collected data from 34 participants. Each participant would view three exam questions with each kind of visualization formats. Each exam question had both search fact task and inference generation task. Hence, the sample contained 204 observations.
Dependent variables represent the user perceptions. The variable learning comprehension is the percentage of the correct answer rate respectively in search fact questions and inference generation questions. The variable understanding of visualization is the correct answer rate percentage of the multiple selection question to test how participants know the meaning of the visualization format. Finally the variable perceived learning is the average score of the measurement items with 5-point scale (strongly disagree to strongly agree).
Independent variables reflect the factors which would have influences on the user perceptions. The variable goal orientation represents the degree of individual is motivated by the opportunity to develop and master new skills rather than desire to demonstrate their abilities or to avoid failure. We transform the original average score of 8-items measurement with 7-point scale to categorical variable which have three values: low, middle, high. The three values represent the score which is under the 25%, between 25% and 75%, above 75% respectively. The variable format is a categorical variable, which takes on values of the format for the graph visualization (line, bar and radar). The variable format is also a categorical variable, which represents the types of the learning task (search-fact and inference-generation).
The control variables include differences between programming-experienced students and beginners. Experience is a dummy variable reflecting whether the students is experienced in programming. We define the students who have more than one year
‧
experience in programming as programming-experienced students. Even though we select the question which have the close class average correct answer rate for each exam, we still control for the effects of exam number, which may has effects on the perceptions of the students. The gender is also controlled. Table 5.4 lists the description and summary statistics of variables. Table 5.5 reports the pair-wise Pearson correlation coefficients of our dependent and independent variables. Table 5.6 shows the frequency table for the categorical control variables.
Table 5.4 Descriptive statistics of variables (N=204)
Variable Mean Std. Dev. Min Max
Dependent variables
Learning comprehension 0.846 0.217 0 1.000
Understanding of Visualization 0.611 0.249 0 1.000
Perceived learning 3.69 0.799 2.00 5.00
Independent variables
Goal orientation Categorical variable:
high(N=54), middle(N=96), low(N=54)
format Categorical variable:
radar(N=68), bar(N=68), line(N=68)
task Categorical variable:
searchfact(N=102), inference(N=102)
Table 5.5 Correlation of variables
(1) (2) (3)
Learning comprehension 1
Understanding of Visualization -0.0023 1
Perceived learning 0.0278 0.2373 1
‧
Table 5.6 Frequency table of categorical variables
Variable Count
When the participants were reviewed questions on TCAV in lab experiment, meanwhile, an eye-tracker were adopted to collect data. A Tobii X2-60 eye tracker with a sampling rate of 60Hz was used to collect eye movement data throughout the experiment. We focused on the following aspects to analyze the eye tracking data:
Area of Interest (AOI): The predefined region in the specific interface where user looked at while performing task. It helps to extract metrics from the selected region.
Fixation: The moment that the eyes are relatively motionless and fix on a portion of the interface.
Saccade: The eye movements occur between fixations.
Transition: The saccade between two AOIs.
In the present study, we defined 4 major AOIs: Visualization, Question, Legend and Title (Figure 5.1) on the visualization interface of TCAV. Two metrics from fixation data, fixation duration and fixation count, were used to show how much time and attention the participants spent on the AOIs. Fixations could reflect user’s attention