VIBRANT: A Brainstorming Agent for Computer Supported Creative Problem Solving

M. Ikeda, K. Ashley, and T.-W. Chan (Eds.): ITS 2006, LNCS 4053, pp. 787–789, 2006. © Springer-Verlag Berlin Heidelberg 2006

VIBRANT: A Brainstorming Agent for Computer

Supported Creative Problem Solving

Hao-Chuan Wang1,3_{, Tsai-Yen Li}2

, Carolyn P. Rosé3 , Chun-Chieh Huang2_{, and Chun-Yen Chang}1 1 _{National Taiwan Normal University, Taipei, Taiwan}

2 _{National Chengchi University, Taipei, Taiwan} 3 _{Carnegie Mellon University, Pittsburgh, PA, USA}

{haochuan, cprose}@cs.cmu.edu, {li, g9415}@cs.nccu.edu.tw, changcy@ntnu.edu.tw

Abstract. This paper describes key issues underlying the design of a tutoring

system that brainstorms with students in order to support qualitative problem solving. Cognitively oriented and socially oriented support are enabled by two technologies, namely heuristic-based feedback generation and community-data-driven social recommendation. Formal representations and corresponding automated reasoning procedures for these technologies are introduced.

1 Introduction

Research in the area of Intelligent Tutoring Systems (ITS) has achieved impressive results in improving student learning especially in well defined problem solving do-mains, such as basic algebra (Koedinger et al., 1997) and quantitative physics (Vanlehn et al., 2005). In this paper we begin to consider how to expand the frontiers of this success into areas less touched by ITS research, specifically, the task of more open ended qualitative scientific problem solving, also known as creative problem solving (CPS) that has been attracting more attention in education. Recent work be-gins to approach this area, such as tutorial dialogue systems (Kumar et al., 2006). A tutoring system that brainstorms with students, called VIBRANT (Virtual Brainstorm-ing), is proposed as an instructional tool for science education.

Consider the following sample CPS question: “What are the possible factors that might cause a debris-flow hazard to happen?”, and subsequently, “How could we prevent it from happening?” In our previous work, students were required to answer CPS questions independently without accessing external resources or peer support. Human graders were recruited to score student answers quantitatively using a rubric devised by domain experts (Chang & Weng 2002). However, this operationalization of CPS has previously been used primarily in evaluating particular teaching methods, and the focus was not specifically on how to scaffold students’ idea generation.

This work proceeds to focus on system behavior for supporting idea generation, which can benefit from collaboration (Kraut, 2003). Towards maximizing the effec-tiveness of brainstorming, VIBRANT offers (1) cognitively oriented support provid-ing brainstormprovid-ing feedback and (2) socially oriented support for discussion group formation. Brainstorming feedback is intended as an interface for students to receive a

788 H.-C. Wang et al.

contextualized view of relevant learning resources; while the social support seeks to recommend suitable peers to students for fostering collaborative brainstorming.

2 Overview of VIBRANT

Based on our prior work (Wang et al., 2005), knowledge of solving a CPS task is modeled as a bipartite graph-based formal user profile (fUP), in which the connec-tions between a student’s ideation and explanation are explicitly represented.

Intelligent Support. A formal ontology is constructed serving as the core device for

organizing experts’ CPS ideas and learning resources. The ontology consists of an is-a hierarchy organizing experts’ ideas into several levels of abstraction. Prescrip-tive feedback messages are attached to specific idea nodes at lower levels and cate-gorical nodes at higher levels. Finite state machines (FSMs) are designed to retrieve learning resources such as feedback. In FSMs, the finite set of states, Q, represents the range of the system’s functional behavior, including actions such as check_coverage or move_upward_in_hierarchy, while the finite

alpha-bet, Σ, represents the set of events that trigger transitions from one state to another, such as all_sub_nodes_covered which in this case triggers a transition to a state called get_new_cateogry. Transition functions

δ

:

Q

×

Σ

→

Q

The feedback prepared by the system consists of two parts, a comment and a

tuto-rial. Comments are evaluative texts responding directly to the most recent idea

sub-mitted by the student to the system, while the tutorial is the instruction that directs the student to the next logical focus node, which may either be an idea node or a categori-cal node, selected by the system adaptively based on its model of the student. The use of the is-a hierarchy is considered beneficial for the FSM-based feedback genera-tion. First, the hierarchy of topics provides a basic for supporting a more organized and coherent brainstorming process. The system may select a next focus for tutorial to maximize the students’ local coverage of categorical nodes that have been partially addressed by the students’ idea entries. Second, comments can be fetched strategically at a more generalized level in the ontological hierarchy when a particular idea entry is semantically ambiguous and thus results in low similarity scores as computed using vector-based information retrieval (IR) methods. The strategy may help remedy the insufficiency of IR-based methods for computing semantic similarity and to improve the relevance of system-prepared comments against students’ ideas.

Social Support. Given a collection of historic fUPs created for previous students, we

may re-model the system of fUPs as a tripartite graph with hyperedges. The condition of student p having an idea q and explaining it by reason r can be represented as a hyperedge epqr , which results in a tripartite graph H=(V, E) where V=S

∪

∪

E={epqr}. A variety of analyses can then be computed over the tripartite graph for social structure discovery, such as using local heuristics to extract particular (hyper) edges as cues for social recommendation or applying Social Network Analysis (SNA) methods to incorporate macroscopic and structural information.

VIBRANT: A Brainstorming Agent for Computer Supported Creative Problem Solving 789

3 Evaluation and Current Directions

We conducted an evaluation of VIBRANT feedback generation using a corpus con-taining 61 CPS idea entries from 10 Taiwanese high school students. The with-hierarchy method generates feedback using the aforementioned approach, while the without-hierarchy method does not make use of a category-based brainstorming plan, so that the selected brainstorming focus motivating the tutorial is based purely on information from the student’s most recent contribution. In this case, the comment offered is the comment attached to the most similar node in the domain model.

Two independent judges were recruited to rate the relevance of each feedback mes-sage prepared by each of the two versions for every idea entry in the corpus quantita-tively. Thus, each coder assigned a relevance score to 122 feedback messages. The result reveals no significant difference on average between the two methods. Fur-thermore, there was no significant correlation between the relevance scores assigned by the two coders, which casts doubt on the reliability of the evaluation metric. Nev-ertheless, the trend was in favor of the with-hierarchy approach. Furthermore, the standard deviation of scores across student entries was lower for both coders in the with-hierarchy approach, which indicates that the with-hierarchy approach may be more consistent in its quality. This makes sense given that in some cases local infor-mation is sufficient for generating meaningful feedback, while other times context is helpful. In our current work we are refining our operationalization of the notion of “relevance”. Follow-up evaluations are planned in the near future.

Moving forward, our plan is to employ VIBRANT as a test-bed for conducting in-tervention studies and further testing social psychological hypotheses in educational contexts, including verifying and measuring the benefits of group brainstorming.

References

1. Chang, C-Y., Weng, Y-H. (2002). An Exploratory Study on Students’ Problem-Solving Ability in Earth Sciences. International Journal of Science Education, 24(5), pp. 441-451. 2. Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent

tutor-ing goes to school in the big city. International Journal of Artificial Intelligence in Educa-tion, 8(1), 30-43.

3. Kraut, R. E. (2003). Applying Social Psychological Theory to the Problems of Group Work. In J. Carroll (Ed.), HCI Models, Theories, and Frameworks (pp. 325-356). NY: Morgan-Kaufmann Publishers.

4. Kumar, R., Rose, C., Aleven, V., Iglesias, A., & Robinson, A. (2006) Evaluating the Effec-tiveness of Tutorial Dialogue Instruction in an Exploratory Learning Context. Proceedings of International Conference on Intelligent Tutoring Systems (ITS 2006).

5. Wang, H.-C., Li, T-Y., & Chang, C-Y. (2005). A user modeling framework for exploring creative problem-solving ability. Proceedings of 12th International Conference on Artificial Intelligence in Education (AIED 2005), 941-943.

6. Vanlehn, K., Lynch. C., Schulze, K., Shapiro, J. A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: five years of evaluation. Proceedings of 12th International Conference on Artificial Intelligence in Edu-cation (AIED 2005), 678-685.