ConceptGuide: Supporting Online Video Learning with Concept Map-based Recommendation of Learning Path

(1)

ConceptGuide: Supporting Online Video Learning with Concept Map-based Recommendation of Learning Path

Chien-Lin Tang

^∗

kitty819231@gmail.com National Chiao Tung University

Hsinchu, Taiwan

Jingxian Liao

^∗

jxliao@ucdavis.edu University of California, Davis

Davis, CA, USA

Hao-Chuan Wang

hciwang@ucdavis.edu University of California, Davis

Davis, CA, USA

Ching-Ying Sung

cysung97@uw.edu University of Washington

Seattle, WA, USA

Wen-Chieh Lin

wclin@cs.nctu.edu.tw National Chiao Tung University

Hsinchu, Taiwan

ABSTRACT

People increasingly use online video platforms, e.g., YouTube, to locate educational videos to acquire knowledge or skills to meet personal learning needs. However, most of existing video platforms display video search results in generic ranked lists based on relevance to queries. The design of relevance-oriented information display does not take into account the inner structure of the knowledge domain, and may not suit the need of online learners. In this paper, we present ConceptGuide, a prototype system for learning orientations to support ad hoc online learning from unorganized video materials.

ConceptGuide features a computational pipeline that performs content analysis on the transcripts of YouTube videos retrieved for a topic, and generates concept-map-based visual recommendations of inter-concept and inter-video links, forming learning pathways as structures for learners to consume. We evaluated ConceptGuide by comparing the design to the general-purpose interface of YouTube in learning experiences and behaviors. ConceptuGuide was found to improve the efficiency of video learning and helped learners explore the knowledge of interest in many constructive ways.

CCS CONCEPTS

• Human-centered computing → Graphical user interfaces;

Information visualization; • Applied computing → E-learning.

KEYWORDS

Education/Learning; Information Seeking & Search; Visualization;

Concept Map

ACM Reference Format:

Chien-Lin Tang, Jingxian Liao, Hao-Chuan Wang, Ching-Ying Sung, and Wen- Chieh Lin. 2021. ConceptGuide: Supporting Online Video Learning with Concept Map-based Recommendation of Learning Path. In Proceedings of the Web Conference 2021 (WWW ’21), April 19–23, 2021, Ljubljana, Slovenia.ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3442381.3449808

∗These authors contributed equally to this work

This paper is published under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution.

WWW ’21, April 19–23, 2021, Ljubljana, Slovenia

ACM ISBN 978-1-4503-8312-7/21/04.

https://doi.org/10.1145/3442381.3449808

1 INTRODUCTION

As the development and accessibility of Internet has reached the states of maturity, people now may access all different educational contents on specialized online learning platforms such as MOOCs, or generic video-sharing platforms like YouTube. Also, when in- person instruction becomes restricted or infeasible due to various personal and public constraints (e.g., in situations of prevalent quarantine due to epidemics), online video learning becomes a natural alternative.

Compared to typical lecture videos on MOOCs with a classroom- teaching-like format and structure of instruction, knowledge-offering or learning videos available on YouTube have their unique appeal to knowledge seekers or learners. YouTube videos offer numerous ways in which viewers may engage in learning processes that are self-paced and socially engaging [19]. YouTube’s learning videos also tend to possess a good range of diversity in terms of topics, for- mats and scopes due to the open and social nature of the platform.

YouTube could cover emerging topics efficiently, and it’s arguably difficult to match the quantity and diversity of YouTube videos with conventional lecture videos produced by regular teachers in schools or MOOCs. Generic video-sharing platforms like YouTube has the potential to provide an elastic and liberal medium for knowledge sharing and instruction [5, 16, 32], where anyone who wants to teach can teach and anyone who wants to learn can access the contents at no or little cost.

While YouTube learning videos provide rich contents and chan- nels for learning, the challenges for YouTube learners are also obvious. Learners will likely need to have certain levels of background knowledge to find, filter, and organize the video materials retrieved [14]. Also, they will need to orient themselves to form appropriate video watching sequences and to pace their own learning, such as to determine what videos are basic (or advanced) and when to watch which video. Novice learners thus tend to face a paradoxi- cal situation where they need the domain knowledge they don’t have in order to consume information available in a generic video archive like YouTube [2, 4, 30].

To deal with the aforementioned issues in content navigation and processing, we noted the potential utilities of concept maps, which were originally developed as a strategy for visual learning and assessment [6, 20, 28, 31], on providing a visualization-based solution for the current issues. A concept map organizes and represents domain knowledge as a graph, of which nodes are concepts

(2)

Figure 1: Overview of ConceptGuide. It provides Concept Map (A), Video Ring (B), Concept Path (C), and Video Path (D) to learners. ConceptGuide shows basic concepts and concept map for the keyword searched (“regression” and "machine learning"

here) by analyzing and organizing videos searched from YouTube interface. When the mouse hovers over one concept “logistic regression”, a pop-up window shows most relevant videos to the concept or subtopic (B), a recommendation path for learning the concept and its prerequisites (C), and the corresponding video recommendation path (D).

and links denote relations between concepts. Previous studies have shown that either building concept maps or learning with the aid of constructed concept maps could benefit learners’ learning processes [6, 13, 26]. Concept mapping provides a simple representation of a knowledge domain, which allows learners to selectively focus on key concepts of the domain. By exercising how to organize concepts into structured concept maps, learners may receive the learning benefits to better remember and recall the core ideas [26]. More- over, by offering an overview of learning contents and showing prior or further concepts to learn, concept maps were also found to help learners locate their current context and reduce the feeling of disorientation [6, 12, 13].

In this paper, we propose a system prototype, ConceptGuide, that supports learning from YouTube videos, especially for those who are novices in a topic (see Figure 1 for a screenshot of ConceptGuide).

After users enter a topic (keyword) to learn, the video search results are computationally analyzed, organized, and visually represented in a concept map-based graph visualization. Users can actively explore relationships between concepts and learning sequences of concepts, check and watch their corresponding videos and previous viewers’ sentiment analysis through the visualization. Furthermore, there are recommendation paths for both systematic and organized learning. The system provides users with multiple, diverse video options for one concept as well.

We conducted a user study to evaluate ConceptGuide. Learning behaviors and experiences of 16 online learners were analyzed.

The experiment results show that it is easier for learners to find videos to fulfill their learning needs by using ConceptGuide than using a general-purpose YouTube search. In addition, the system helps them learn systematically, which enhances motivation and

learning efficiency. We also asked the learners to assess the usability, rationale, and comprehensibility of the tool design. The feedback shows that ConceptGuide provides a clear conceptual model and is functionally useful for learning from YouTube learning videos.

The paper contributes to the research and practitioner communi- ties through three different angles: (1) Identifying an integral computational pipeline and interface design that provide a structural view of video learning contents orienting toward concept-based learning; (2) Developing a scalable web system prototype for concept map generation and learning path recommendation, enabling user testing and evaluation;(3) Demonstrating a proof-of-concept, showing the effectiveness of concept map-based video recommendations in guiding learners to visit more, and more diverse, concepts of a domain, and to engage in learning.

2 RELATED WORK

2.1 Interactive Interface for Online Learning

Previous work have proposed methods for visual analysis of educational data and novel visualization features in order to improve students’ learning efficiency. For example, several systems [11, 18, 36, 37] help users navigate through single instructional videos by providing various visual designs to show keyword summary, points of learners’ interest, or a video’s topic. Fraser et al.

introduced application-independent approach to search segments of videos by keywords, speech and in-video pointing [8, 9]. Zhao et al. [39] integrated visual, audio and textual information of a video and presented them with novel visualization components to help users explore an educational video. Mahapatra et al. proposed automatic hierarchical table of contents and phrases for the user

(3)

to preview the content of a video [23]. Kim et al. [17] proposed an interactive system that provided multimedia exercises embedding in lecture videos. It helped students easily respond to teachers by using videos, audios, and inking and thus increased students’

engagement. However, aforementioned research focused on improving learning or navigating efficiency of a single lecture video, which did not provide learners an overview across multiple videos.

In contrast, some research focused on improving learners’ searching and learning efficiency from multiple educational videos and other material pools. Adcock et al. [1] presented a search engine for more than 10, 000 videos of classroom lectures by using OCR technology to recognize the content of slides presented by teachers during the classes, in which users could enter terms of interest to them and find specific contents of videos. Fatiha et al. [3] proposed a system which recommended appropriate MOOC courses for learners by analyzing their profiles, learning needs and prior knowledge. However, systems illustrated above did not consider the relationship between videos/courses and the learning sequences which might cause fragmented learning.

2.2 Concept Map Construction

Concept map is a well-known and widely used tool in education.

It provides a visual representation to denote concepts and inter- concept relations as a human-comprehensible graph. Concept map is commonly used to represent and organize knowledge by instructors and/or students, and has been applied to online education [25].

Shaw [31] investigated learning performance of using concept maps to organize e-learning materials. The result shows that learning efficiency can be improved and learners may acquire knowledge better with the aid of concept map representation. Concept maps are typi- cally constructed manually, and thus construction cost tends to be high and quality may depend on individuals. While manual construction of concept maps remains to be of great value in learning assessment for understanding learners’ knowledge states, there’s limited applicability in online instructions such as helping learners learn from watching online videos.

Research interest in automatic concept map construction has grown in recent years. Several techniques investigated extracting useful features from educational data to build concept maps. Lee et al. [20] extracted key concepts based on the Term Frequency- Inverse Document Frequency (TF-IDF) method to create concept maps for text data. Wang et al. [34] collected proper nouns from Wikipedia Glossaries as concepts to build concept maps from transcripts. Wang & Liu proposed a two-phase model for prerequisite concept map for teachers which includes domain concepts extraction and prerequisite relationships identification [35] for the purpose of assessing students’ learning performance. Zhao et al. [40]

used Latent Dirichlet Allocation (LDA) to obtain possible topics from lecture videos. Pan et al. [27] focused on detecting prerequisite relationships between concepts by using MOOCs data. While these works are diverse in their technical approaches, they share the goal to explicate the conceptual structures embedded in raw contents for supporting applications and human understanding.

2.3 Concept Map-based Visual Interface

Since the advantages of concept map have been well-noted in recent years, researchers started to apply concept maps to their educational systems. Schwab et al. [30] constructed an interactive education system with a concept map showing learners’ personalized learning path. The system also provided an interface for instructors to construct concept maps by themselves and upload learning materials for each concept. Liu et al. [21] presented ConceptScape, a system that generates interactive concept maps by prompting learners’

reflections while watching an educational video. This system had both concept maps and corresponding time anchors linked to video segments, aiming to contextualize in-video navigation with concept map as the conceptual context. Zhao et al. [40] proposed a video recommendation system called MOOCex. It automatically analyzed topics and relationships between videos and then visualized the semantic information onto a 2D semantic space for MOOC learners.

The aforementioned systems [21, 30, 40] constructed concept maps mostly based on structured educational data such as MOOCs data and the scope of application is also limited to instructors and learners with a relatively clear plan of learning (e.g., a clear syllabus to follow). The former systems did not consider learners’ feedback in their systems. However, due to the lack of syllabus and other course information, it becomes challenging to extract relationships among unstructured learning videos on YouTube automatically.

3 METHODS

3.1 System Overview

ConceptGuide provides keyword-driven interactive navigation of video contents for learners to explore videos uploaded on YouTube, as shown in Figure 1. It searches YouTube videos based on the keyword or topic given by users and returns a concept map by analyzing video transcripts, along with recommended videos and suggested learning sequences according to the relevance of the keyword to associated concepts in the map, quality of videos, and feedback commented on the video on YouTube. As shown in Figure 1, ConceptGuide has four views on the interface: (A) Concept Map, (B) Video Ring, (C) Concept Path, and (D) Video Path.

3.2 Interface Design

For an input query, a concept map is constructed from top YouTube videos by using a system workflow that we’ll describe later. We apply the force-directed graph [10] to visualize the map. Each node in the concept map represents a concept and its radius represents the term frequency of the concept. The edge distance between any two nodes is inversely proportional to the similarity between the two nodes (concepts) where closer nodes are conceptually more relevant. Users can click on a node to see more details about each concept. Figure 1(A) shows that when users hover over the concept

“logistic regression”, interconnected concepts are highlighted in blue and recommended learning paths are also denoted with solid lines and arrows. When a user clicks on the node on the Concept Map, a pop-up window with Video Ring, Concept Path, and Video Path displays detailed information and suggestions for learning.

Video Ring (Figure 1(B)) displays the videos for the concept chosen by the user with rich information, including the frequency

(4)

of concept appearance in the video and the positiveness of feedback from previous viewers.

Concept Path (Figure 1(C)) provides a concise summary of the selected concept and its relation to other concepts in the concept map, such as prerequisites and follow-up concepts. It’s intended to offer learners a clear concept-based learning path to follow. The gray lines denote the prerequisite relationship between concepts. Nodes on the Concept Path are also selectable and the corresponding parts in other views will also be updated simultaneously.

Video Path (Figure 1(D)), on the other hand, specifies the videos for Concept Path to help learners. Since every video includes multiple concepts, we also modify the ordering of these videos for better learning performance. Users can hover over a video to check their corresponding concepts (highlighted) in this part, too. Moreover, the background colors of the order number represent the video’s sentiment (nodes coded in darker green for more positive feedback from viewers).

3.3 Concept Map Construction

Figure 2 shows an overview of the construction of concept map.

The system first collects the videos by YouTube Data API [33]

and gets speech transcripts using Python package youtube-dl [38].

Our system keeps videos that satisfy two conditions: (1) with Eng- lish captions (2) length less than 20 minutes. These conditions are selected experimentally considering typical video duration and re- sponse time of the system. Usually, there remain about half of the searched videos for each query. And, the comments and the number of likes/dislikes of each video are used to analyze the sentiment of learners’ feedback, which may imply the quality of a video. Con- ceptGuide also utilizes other video information such as thumbnails, links, duration, and title, with a proper visual encoding to present the concept map and recommend learning path and video ring in the interface.

We adopt NLP techniques (described below) to automatically generate a concept map showing the structure of knowledge. We collected main concept words by extracting keywords directly from transcripts, applying Wikipedia glossaries, and using videos’ tags.

Then the relationship between concepts, including similarity and prerequisite is analyzed to construct the map. Some linguistic rules and features about concepts are applied with weights according to the literature and training. All the weights in ConceptGuide are tuned with our current dataset, consisting of dozens of topics spanning across a few different fields.

3.3.1 Concept Extraction.We first apply rapid automatic keyword extraction (RAKE) algorithm [29] to collect keywords directly from transcripts. RAKE determines keywords based on the frequency of word appearance and its co-occurrence with other words in the document. After collecting keywords, we solved two semantic issues: detection of domain-specific keywords and identification of polysemies.

Inspired by Wang et al. [34], we applied Wikipedia glossaries to filter out irrelevant keywords in our system. Wikipedia glossaries include key concepts which have Wikipedia pages in the same domain, which implies the importance of these concept words. For the purpose of prototyping, the researchers selected 37 different Wikipedia top-level glossaries covering many different fields, such

Table 1: Five features for determining the importance of a concept.

Features Description

Tf term frequency of the concept appearing in all searched videos

MaxTfidf the maximal tf-idf of the concept among all videos

IsInTitle one if the concept appears in the title of any searched videos and zero otherwise IsMultiWords one if the concept contains multiple

words and zero otherwise

NumOf Videos the number of the videos containing the concept

as architecture, calculus, chemistry, history, etc. For a better performance of Wikipedia glossaries filter, we used Google Cloud Natural Language (Google NLP) API [15] to detect the domains of videos before the application of Wikipedia glossaries. The relevant Wikipedia glossaries within the domain of the videos are applied.

The Google NLP API helps identify the domain of videos so that words with polysemy (i.e., with multiple meanings) but having a positive meaning in the context (i.e., in their video’s domain’s Wikipedia glossaries) are retained.

Besides the transcripts, we also used the tags of the YouTube video to extract concepts. They often contain useful information of an educational video from its uploader’s perspective, such as field, specific technique, or key concepts.

Wikipedia glossaries, keywords from transcripts and tags are integrated to form a long list of concept words. They are converted to lower case and lemmatized. For visualization, we sorted these concept words by their importance, which is calculated from the features listed in Table 1 and chose top 30 important ones for the concept map.

To determine the relative importance of concept words, we chose to use five features to select the most important concepts from the long list of concepts extracted, listed in Table 1. Features Tf and NumOf Videosconsider the occurrence of the concept in videos.

MaxTfidf is adopted because of its meaningfulness for individual videos. Video’s title is often the most important information. Fur- thermore, proper nouns often appear with two or more words, so we take IsMultiWords into account. All of these features are normal- ized to range [0, 1] with selected weights: Tf =0.2, MaxTfidf =0.6, IsInTitle=0.25, IsMultiWords=0.4, NumOfVideos=0.45.

3.3.2 Concept Relationship Analysis.To construct a concept map, the relationships among concepts, including their similarity and prerequisite are analyzed. Concept similarity represents the distance between concepts: similar concepts are closely related in semantics. We use N-gram method to form concept vectors for semantic distance. And prerequisite relationship decides the direction of learning path. We compute the similarity and prerequisite relationship for every possible concept pairs (a,b) as follows.

Concept Similarity.To measure the similarity between two concepts, we consider both local and global similarity, such as the co-occurrence of the two concepts in the same video Vr and in

(5)

Figure 2: Overview of concept map construction

WikipediaW ikiRe f .Vr(a,b) is the cosine similarity of the tf-idf vector of two words in the video. W ikiRe f (a,b) denotes whether the conceptb is referred in the concept a’s Wikipedia page;W ikiRe f (a,b) = 1if referred.

Prerequisite Relationship.We have four concept features to infer the prerequisite relationship between concepts, which were proposed first by Pan et al.[27]. These features are Semantic relatedness (Sr), Video reference relatedness (Vrr), Wikipedia reference relatedness (Wrr)and Complexity level distance (Cld). They cover semantic, contextual, and structural aspects of prerequisite relationship [27].

Semantic relatedness (Sr)represents the semantic closeness between concepts from their frequency in all videos and their occurrence in other concepts’ Wikipedia page. We adopt the same definition of concept similarity to compute Sr since they both measure similar relationship.

Video reference relatedness (Vrr)uses the difference of frequency of keywords in the videos to get contextual information about simple videos/concepts and advanced videos/concepts. Vrr follows a simple phenomenon that basic concepts are mentioned frequently in videos of earlier lectures while advanced ones are seldom mentioned in earlier lectures [27].

Vrr infers the prerequisite relationship of two concepts a and b by calculating their term frequency in all videos. As listed in Equation 1, Vrw(a,b) quantifies the ratio of concept b referred in the videos that mention concept a. And GVrw(a,b) considers the possible sparsity of video concepts and weights Vrw(a,b) by videos related to concept a, rather than mentioning a precisely [27].

If Vrr(a,b) > 0, it means concept a is a prerequisite of concept b from the Vrr aspect.

Vrr(a,b) = GVrw(b, a) − GVrw(a,b), GVrw(a,b) =

Í_M

i=1Vrw(ai,b) · w(ai,b) ÍM

i=1w(a_i,b) , Vrw(a,b) = Í

v ∈V f (a,v) · r(b,v) Ív ∈V f (a,v) ,

(1)

where a1, ..., a_M ∈C are the top-M (M=10) most similar concepts to concept a and C the set of all concept words. And w(a,b) is the cosine distance between vaand v_b, where vais a vector that represents a concept a, obtained by the word embedding method Word2Vec [24]. Furthermore, V is the set of the searched videos, f (a,v) is the term frequency of concept a in video v and r(b,v) is one if concept b appears in video v and zero otherwise.

Wikipedia reference relatedness (Wrr)is selected due to the similar idea of Vrr. That is, basic concepts are mentioned more often in the Wiki pages of advanced concepts. Thus,W rr implemented the same method of Vrr on Wikipedia pages to calculate the relationship from the context of Wikipedia.

Complexity level distance (Cld)is modified from Pan et al.’s paper [27] to compare total coverage of concepts. A basic concept is more likely covered in more videos or survives longer time in a course than the advanced ones. It originally considers average video coverage (avc) and average survival time (ast) of a concept to measure the complexity level between two concepts when video sequences are available in Pan’s paper. In our implementation, we

(6)

simply compute Cld as the difference between the average video coverage of two concepts, Cld(a,b) = avc(a)−avc(b), where avc(a) is the ratio of videos in V mentioning a.

We compute the weighted sum of these features as prerequisite score P(a,b) after normalizing the value of each feature into [0, 1]. If P(a,b) = 0, there is no prerequisite relationship between concept a and b. If P(a,b) > 0 , a is b’s prerequisite. In contrast, if P(a,b) < 0, b is a’s prerequisite. As Pan et al. [27] suggest that Sr and Cld are the most useful features for detecting prerequisite relationships, the weights of Sr and Cld are both set as 3 while the weights of the others are set as 1.

3.4 Learners’ Feedback Analysis

Previous learners’ feedback on a video is one useful metric that measures video quality. We analyzed previous video viewers’ feedback of a video by considering the sentiment of their comments and the numbers of likes/dislikes for the video. We visualize it into the Video Ring and Video path. We adopted TextBlob [22], a natural language processing toolkit in Python to analyze the sentiment score of each comment. Specifically, we compute the learners’ feedback F (v) of a video v as follows,

F (v) = w1· S(v)

S_max(v) +w2·L(v) − D(v) L(v) + D(v),

where the first term denotes the sentiment score of comments of the video and the second term denotes the preference of the video. S(v) is the average sentiment score of the all comments on video v. Smax(v) is the maximum value of S(v) among all searched videos. L(v) and D(v) are the number of likes and dislikes of video v, respectively.

3.5 Video Visualization Selection

Figure 3: Concepts are highlighted while user’s mouse is over a video (the second video here)

Concept Path presents a full list of concepts which have prerequisite relationship with the selected concept word. These concepts are highlighted on the concept map, too. Figure 3 illustrates the highlighted concepts in a concept map and the concept path when users mouse-over a video. We use topological sort [7] to transform this small directed graph of the highlighted concepts into a linear ordering of its vertices with the most basic concept at the top.

Video Path is designed to provide a sequence of recommended videos for study. First, our system selects a representative video of

each prerequisite concept by considering the relevance and feedback of videos. The first related video whose sentiment score is positive would be selected as the representative video. Then, a proper video watching sequence is determined by calculating video’s average appearance frequency in the concept path.

Video Ring visualizes the videos that are most relevant to a selected concept with rich information, like frequency of the concept mentioned in the video and previous viewers’ feedback. As showed in Figure 1(B), small circles with video snapshot have different radius and colors. The radius of each circle represents the frequency of the concept appearing in this video. And from the largest to the smallest, these circles are placed in a clockwise direction starting from the twelve o’clock position. The colors of outer rings show the emotions of comments on the video. We use green for positive and red for negative feedback. The legend is shown on the right side of view (B). Users can hover over these circles to check video’s title or click them to watch the video on YouTube. Other information of a video, such as title, thumbnail and duration are also displayed within the circle. Furthermore, YouTube icon in the center of the grey circle links to YouTube search page of the concept.

The code, demo, and more details of ConceptGuide is shared in https://jxliao6.github.io/ConceptGuide.github.io/.

4 CASE STUDIES

We demonstrate three cases of ConceptGuide on different learning topics, including “sorting data structure” and “natural language processing” and “COVID-19”. This section describes the processes of searches on ConceptGuide and discusses our findings.

4.1 Case: Sorting Algorithms

Figure 4: The Concept Map View of “sorting data structure”.

Here unsorted list is clicked and its related concepts are highlighted (shown as blue nodes).

Figure 4 shows the result of the system-generated visualization of concept map for “sorting data structure” as the keyword inquiry.

Concepts of several main sorting algorithms are identified and

(7)

listed by ConceptGuide, including merge sort, insertion sort, bubble sort, quick sort and selection sort. Also, the system identified close connections between the concepts of sorting algorithms and the concepts of data structures in this example, of which terms such as sorted array, element, position, increasing order are also extracted, along with description for these terms. Time complexity is another important aspect identified for this sorting concept. Nodes of different concepts are shown on the map are designed to be clickable.

Figure 5 shows the Video Rings for the conceptual nodes corresponding to different sorting methods. The videos of each Video Ring precisely introduce the corresponding algorithm, showing satisfactory system accuracy in this case. Most importantly, the result demonstrates the system’s capability to analyze and present video search results with an easy-to-follow visual organization.

Figure 5: The Video Ring of different sorting algorithms (insertion sort, quick sort, selection sort).

4.2 Case: Natural Language Processing

NLP is one of the learning topics we used in the experiment. We’ll illustrate its concept map (Figure 8(d)) and the results of evaluation about this case in the result section. Figure 6 shows the Concept Path of the conceptual node text analytics when a learner clicks on it on the concept map. Learners can see and visit several prerequisite concepts of text analytics from the view. For example, concepts listed before text analytics such as bag of word, name entity and sentiment analysisare all common techniques used for analyzing text data.

Figure 6 also illustrates two examples when a learner hovers on the third and the fourth videos of the Video Path. The concepts taught in each video are highlighted with red background. The third video (left of Figure 6), which talks about name entity, teaches learners how “name entity recognition” is performed in NLP. “Name entity recognition” is a common procedure to perform after tokenization in text analysis. On the other hand, the fourth video (right of Figure 6) extends text preprocessing to sentiment analysis, which is a further application to classify the polarity of a given text or analyze the attitude of a writer. This case demonstrates our system’s capability to organize and recommend videos to learners hierar- chically, covering basic to advanced concepts. It also shows the system’s utility on guiding learners to visit these concepts step by step, starting from relatively more basic and fundamental concepts to advanced ones.

Figure 6: The Concept Path and Video Path of text analytics with mouse hovered on third video (left) and fourth video (right). Concepts that are taught in the hovered video are highlighted in red background.

4.3 Case: COVID-19

COVID-19 is an emerging learning topic on YouTube. Figure 7 shows the concept map generated by using “COVID-19” as the searched keyword in September 2020. Concepts extracted from different COVID-19-related videos are identified, such as symptoms, latest research evidences, precautions, treatments, and news. As scientific concepts, Cell membrane and immune system are closely related on the concept map with the official name of the new coron- avirus, SARS-CoV-2. It also captured different keywords commonly appeared in those news videos, such as social media, conspiracy theory.

As mentioned in the introduction, YouTube videos are appealing to knowledge seekers due to the diversity of contents, and efficiency of content production as demonstrated in this case. This COVID-19 concept map extracted main concepts out of different videos, and even some of which are not learning oriented. The results show that the system could help users to filter and integrate diverse and unstructured videos, and providing an initial systematic plan (even with some imperfections) for learners to understand about the topics, especially for emerging topics.

5 EXPERIMENT

To evaluate the performance of ConceptGuide on learning assis- tance, we invited participants to learn new topics on both YouTube interface and our ConceptGuide platform, to collect their comments and feedback about ConceptGuide. We had a full questionnaire and interview with participants.

5.1 Participants and Procedure

We recruited 16 participants (9 females and age 20-25) from a college.

50% of the participants major in engineering-related fields, and the others major in business, social science, health, and other fields.

Also, 15 of 16 participants had learning experiences on YouTube before. The experiment was a within-subject experiment to control individual differences in learning. Each participant was instructed

(8)

Figure 7: The Concept Map View of “COVID-19” searched in September 2020. “United States” is clicked and the related nodes are highlighted in blue color.

to complete two learning tasks, one with YouTube and another with ConceptGuide, respectively.

The two tasks were taken at least one hour apart to prevent tiredness. We first demonstrated an introduction and instruction of ConceptGuide and then gave participants five minutes to get familiar with the system. The instruction of learning tasks is “Please search and watch related videos on [system] to learn about [topic]

as much as you can.” Here [system] could be either YouTube or ConceptGuide, and [topic] was chosen from Bitcoin and NLP. The order of [system] and [topic] was counterbalanced to mitigate possible ordering effects in the within-subject design.

We chose these two topics for the study because there are rich YouTube video resources available for them online. And from our pilot survey about the participants, they were interested in learning these new topics. This ensures the original motivation of learning, mimicking real online learning scenarios. The two topics are also very different so that the learning of one is independent on the other. We restricted the study time for each video in 50-70 minutes.

At the end of the experiment, each participant took a questionnaire and interview about their experiences of using ConceptGuide.

In our experiment, we compared the learning results and experiences in two different learning systems: YouTube only (C1) and ConceptGuide system (C2). In C1, without any concept map, participants could only use YouTube to search and watch videos. In C2, participants used ConceptGuide mainly. The concept map and other information obtained with keyword “Bitcoin” or “NLP basic knowledge” were provided with ConceptGuide system. Participants were allowed to directly search other videos on YouTube if they were not satisfied with those videos provided by our system.

5.2 Metrics

Questionnaire.We designed a questionnaire with 23 questions which covered seven aspects to compare participants’ learning experiences and attitudes between two conditions, such as learning concentration, usability of system, learning motivation, scope of videos, quality of videos, learning guidance and self-perceived learning performance.

Interview.At the end of the experiment, we also took a in-depth interview with the participants about their learning experience.

The questions focused on three aspects: (1) searching and learning procedure during the task; (2) the influence due to our learning system; (3) feedback on different components of ConceptGuide.

Users’ clickstream and browsing history.We collected all the user behavior via logs during the experiment, including participants’ searching strategies, types of videos they chose, keywords they used to search on YouTube as supplement, time they spent on searching or watching videos, etc.

6 RESULTS AND ANALYSIS 6.1 Manipulation Check

Concept maps are automatically generated by ConceptGuide from YouTube videos. We checked the validity of the concept maps with concept words visited and materials consumed by learners in both conditions to understand the breadth and relevance of the concepts recommended to learners by ConceptGuide.

We retrieved learners’ histories of video watching and marked down those concept words simultaneously present in the automatically generated concept map and appeared in the videos watched in the two conditions. If the video recommendations generated by ConceptGuide weren’t good enough to support learning, we would expect to see the two conditions resulting in similar concept coverage as poor recommendations shouldn’t benefit learners and may instead interfere with them.

Figure 8 showed the results of concept word coverage for learning the topics of Bitcoin and NLP. From Figure 8, we can see that the concept coverage of C1 overall appeared to be smaller than C2 (i.e., the condition with ConceptGuide) regardless of the learning topic. Specifically, for each concept word, we counted the numbers of participants who have visited it. And we summed up these numbers across all concepts. There were 288 and 381 times of concept learning (i.e., visit of a concept) in C1 and C2, respectively. This again shows that participants in C2 encountered and visited more concept words. Participants not only visited the core concepts of the topic but also more related concepts recommended by Concept- Guide. It shows the benefit of our system as a visual prompt for learners to discover more aspects of the topic.

6.2 Questionnaire and Interview

Figure 9 shows the results of the questionnaire. They represent participants’ evaluations about ConceptGuide. Repeated-measure ANOVAs found that mean scores of ConceptGuide (C2) were significantly higher than those of YouTube (C1) in five of the seven aspects of evaluation. Combined with corresponding interview feedback, the results of learners’ experience with ConceptGuide are reported here.

(9)

(a) C1, Bitcoin (b) C2, Bitcoin

(c) C1, NLP (d) C2, NLP

Figure 8: The concept coverage in C1 (Youtube only) is smaller than that in C2 (with ConceptGuide) regardless of the learning topic. For the same topic, the concept maps are the same in both conditions. Concepts with redder highlight represent that they were visited by more participants in the condition mentioned below within the topic.

Figure 9: Learning self-evaluation of questionnaire. Con- ceptGuide (C2) was significantly better than Youtube (C1) in five of the seven aspects of evaluation. The error bars represent the standard errors of scores.

Learning concentration.The concentration score of C2 (M=5.979, SD=0.5640) was significantly higher (F(1,15)=4.543, p=0.0083) than C1 (M=5.292, SD=0.7084). In the interview, five participants reported they were more concentrated on learning in C2 than learning in C1. Four of them mentioned that high content overlap of videos in C1 caused them getting distracted at the middle of the task. An- other participant reported she got distracted in C1 because of her discontent about the videos.

“Sometimes I felt distracted while watching videos. I kept skimming the related videos on the right side or the comments below to check whether there are better videos.”(P16, C1)

Usability of system.The score of C2 was also significantly higher (F(1,15)=4.543, p=0.0578) than the score of C1. The average score in C1 and C2 were 5.667 (SD=0.7404) and 6.167 (SD=0.7794), respectively. Most participants (n=10) expressed the interface of ConceptGuide was concise and easy to understand. Some other participants (n=2) mentioned that they needed more time to get familiar with ConceptGuide due to lots of information presented in our system, but they both agreed that the information provided in our system was useful.

Learning motivation.The average score of learning motivation in C1 and C2 were 5.125 (SD=0.9728) and 6.042 (SD=0.8767), respectively. The score in C2 was significantly higher (F(1,15)=4.543, p=0.0000) than in C1. Some participants asked to use ConceptGuide to learn other topics such as topics of chemistry or physics when they need a review of knowledge or prepare exams, which shows their willingness to use our ConceptGuide system. It also confirms ConceptGuide’s usefulness to summarize and visualize knowledge of a new domain for learners.

Scope of videos.The score of the scope of learning of C2 (M=6.042, SD=0.8682) was significantly higher (F(1,15)=4.543, p=0.0014) than C1 (M=4.875, SD=1.327). Most participants (N=11) mentioned the problem of the high overlap of videos on YouTube search results (C1). Half of the participants (N=8) reported that our system helped them find different aspects of the topic or the content they had not learned yet.

“...For example, I found the concept “white paper” from the concept map. Maybe it was an important concept of Bitcoin. However, if I search from YouTube (C1) and none of the top-ranked videos mention it, I would probably miss this important concepts.”(P5)

Quality of videos.Video quality impacts the learning experiences ultimately for everyone. The average score in C2 (M=5.417, SD=0.8028) was higher (F(1,15)=4.543, p=0.051) than C1 (M=4.917, SD=0.7149). However, the results of interview are divergent. Seven participants reported that they did not realize an obvious difference in video qualities chosen between two conditions. There are also 7 other participants reported that videos in C2 had better quality than the videos in C1.

For the interface design of concept highlight and sentiment information about video quality, nine participants (out of 16) reported that they checked the sentiment polarity of videos before watching, however, nearly half of them (N=7) said it was not their main con- sideration when choosing videos. It did not affect their decisions unless the feedback was extremely negative. On the other hand, we asked participants whether they checked the highlighted concepts of each video. Seven participants reported that they browsed this information while choosing videos. We founded participants (P8, P10, P13) who used both functions tended to be more satisfied with the videos.

“I first selected the video with the highest sentiment polarity. The quality of this video which shows dark green is really good. ... I checked highlighted concepts of the video when selecting videos. If it highlighted something I am interested in, I would choose this video.”(P8) Learning guidance.The average score of C1 and C2 were 3.484 (SD=1.171) and 5.625 (SD=1.004), respectively. The score of C2 was

(10)

also higher (F(1,15)=4.543, p=0.0001) than the score of C1. Almost all participants (N=14) agreed that our ConceptGuide system helped them learn more systematically. They could keep finding unknown concepts they never heard, and found related videos with more clues. Fifteen participants also mentioned that in C1 they were perplexed for the videos they should watch next sometimes, due to the high overlap of videos. With the help of our system, search efficiency was highly improved.

“The last video I watched summarized several basic concepts of the previous videos. If I haven’t learned those from the previous videos, I probably wouldn’t understand the content of this video.”(P1, C2)

Perceived learning performance.We also asked the participants to self-evaluate their learning performances in the questionnaire. Participants generally felt that they learned significantly better using ConceptGuide (F(1,15)=4.543, p=0.0106). The average score of C1 and C2 were 5.094 (SD=0.9169) and 5.828 (SD=0.8598).

We further analyzed the learning results by interviewing the participants. Their interviews are slightly different from the questionnaire. Half of the participants (N=8) reported that they did not tell the difference in learning performance between C1 and C2 because of the limited time and the difficulty of comparing different learning topics. Most of the other participants (N=7) reported that they learned better in C2 because of the overview and step-by-step guide. It was helpful for them to check whether there were any missing concepts in the map, thus they were more confident about the integrity of learning.

“I feel more confident about the score of posttest after learning with the learning system (C2) than YouTube (C1).”(P1)

“I learn from basic concepts such as “tokenization" to advanced concepts in Natural Language Processing (C2), but the videos listed on YouTube (C1) start from advanced concepts. I prefer to learn step by step, and the learning system (ConceptGuide) meets my learning strategy. Thus I think I learn better by using the learning system.”(P2)

6.3 Browsing Behaviors

Table 2 shows the statistics of browsing history in both conditions, including the time they spent on searching on YouTube/ConceptGuide, time on watching videos, duration of the task, the number of additional keywords they searched on YouTube and the number of videos they watched during the task.

About the time management, we found that ConceptGuide helped participants explore more videos. The average number of videos watched in C2 (10.25) was more than in C1 (9.13), even though the total video watching time in C1 and C2 were nearly the same (53.47 and 54.22). And from the browsing histories, four participants frequently paused the videos or repeated some clips, spending much more time on taking notes while watching videos in C1 than in C2.

In the interview, they explained that they could easily acquire core concepts from ConceptGuide (C2) so they just focused on those concepts and skip unimportant contents.

Another obvious difference between C1 (M=2.88) and Concept- Guide (M=0.13) was the number of additional searches. Only one participant in C2 searched other keywords in YouTube more than once, but participants of C1 searched much more times than those in C2. From the interview, seven participants who kept searching in C1 reported that it’s due to the high overlap of the previous videos,

they needed to find more diverse videos about the learning topic.

Eight participants who didn’t search other keywords in C2 - i.e., only using the initial keyword Bitcoin/NLP - reported that they thought the learning system already provided enough information for novice learners. The results showed that our learning system reduced the times that participants needed to search.

7 DISCUSSION

Learning platforms based on unorganized educational videos like YouTube need to provide appropriate guidelines to help learners find sequences of videos that are not only conceptually relevant, but also feasible for learners to acquire knowledge from the beginning to advanced levels. Through the evaluation study of our system prototype, ConceptGuide, we’ve obtained experience, results and useful feedback that could inform the iteration of ConceptGuide, and design of systems with similar goals.

7.1 Reflections on ConceptGuide’s Design and Engineering

To prototype ConceptGuide, we employed multiple NLP techniques and data sources to enable a computational workflow capable of automatically constructing concept maps from online video contents. We modified the computational techniques to adapt to the constraints or characteristics of educational contents available on YouTube (e.g., lack of structure with new contents added, re- moved and updated constantly). For instance, we took advantage of Wikipedia entries and page contents to assist with the construction of concept maps, offering structures for unstructured contents dynamically found on YouTube. While arguably the concept maps automatically generated by the computational workflow may not be able to match the quality of expert-created concept maps, it is important to note that in this project we aimed to develop a flexible and salable concept map generation mechanism for indefinite future topics. That is, it is not our goal to handcraft a perfect concept map on a very narrow topic to demonstrate “how great it can be”.

Rather the purpose of our prototyping and evaluation work is to investigate what’s “good enough” to generate a noticeable effect of learner’s experience.

ConceptGuide is grounded on a plausible system idea aiming to add structures to support navigation and learning for unstructured contents. We strive for a balance between scalability and accuracy of the concept representation, and currently used the same set of pa- rameters for all learning topics for generating all the visualizations.

Given the positive, constructive results of the current evaluation on the current prototype, we believe that the tradeoff is worth it. As our “good enough” concept maps could lead to positive impact on aspects of online learning, it is expected that any future improvements in the precision and recall of concept map generation would further improve ConceptGuide’s utilities and impact on learning.

7.2 Implications to Supporting Learning Disorientation

Instead of searching and watching videos without clear guide, learners using ConceptGuide could follow the concept map and other

(11)

Time on (mins) Number of YouTube searching ConceptGuide watching videos total experiment searches videos

Learning system ^Mean ^Std ^Mean ^Std ^Mean ^Std ^Mean ^Std ^Mean ^Std ^Mean ^Std

YouTube (C1) 3.13 1.82 - - 53.47 5.11 56.61 5.11 2.88 2.39 9.13 4.63

ConceptGuide (C2) 0.03 0.12 5.75 2.72 54.22 7.14 60.00 6.22 0.13 0.34 10.25 3.19 Table 2: Statistics of distribution of learning time and number of learning videos

scaffolds provided by the system to explore the video contents struc- turally and approach specific learning goals. Throughout the evaluation study, our findings resonate the point of view that structural visualizations offered can guide learners to explore the conceptual space more efficiently, as shown in the results of concept coverage (Figure 8) and efficiency of content search (Table 2).

Furthermore, browsing history and interviews of participants showed plenty of benefits we expected and unexpected. From the interviews, we obtained more clues from users about how they chose learning materials and made use of paths. Overall, people who selected videos directly from YouTube (C1) would choose videos from the top of the video list one by one. And they need much more searches when using only YouTube for learning as shown in Table 2.

They considered features such as video’s thumbnails, the number of viewers and titles to decide whether to watch a video. However, technically those features may not fully reflect the content of online videos. Plus, learners may not have sufficient cognitive capacity (experience, domain knowledge, skills, etc.) to decide what to search and what to watch, resulting in disorientation in an unstructured learning space [2, 4]. On the contrary, ConceptGuide can provide learning scaffold that simultaneously (1) augment learners’ ability to browse and choose relevant contents through visualizations, and (2) reduce the complexity of the media space through learning path recommendations. A combination of the two could effectively reduce learning disorientation without relying on more sophisticated user modeling and personalization techniques as demonstrated in previous work [4].

7.3 Impact on Experience of Learning

Overall, our system provides learners with useful, constructive experiences of learning with video recommendations for the purpose of learning. It improves learners’ attitudes in multiple aspects.

Our system is shown to improve learners’ concentration. Learn- ers could watch the current videos more attentively without having to excessively search for other relevant videos. Their enthusiasm to learn is also largely maintained to help them study broader aspects of the topic. Moreover, they are more confident about their learning outcomes, which is another support of learners obtaining constructive learning experiences from our system. It shows that the design of ConceptGuide may have effects that go beyond simply providing conceptual guidance to navigate the content space, but also may impact learner’s self-efficacy and confidence, which are known to be crucial to exploratory learning, positively.

7.4 Limitations and Future Work

The current ConceptGuide implementation still has some limitations. One search on our system takes about 10-15 minutes currently.

So learner’s learning and experience may suffer from it. To ame- liorate this issue, future implementations may cache the search and analysis results of a keyword. This could greatly shorten the computational time. Also, graph visualizations like concept maps are not necessarily intuitive or easy to learn to different learners.

It’s known that when the number of concepts encoded in a concept map increase, the visual complexity of the map representation might increase exponentially due to inter-concepts links. There may be other interface features useful to decrease the burden of interaction with the map. Given the positive outcomes of the current current prototype, improvements to system performance is likely to further heighten the positive effects we observed now.

Furthermore, experiment results illustrate the impact of Concept- Guide on young adults since all the participants are sampled from a college. To support online video learners with different learning preferences and diverse digital literacy, future system design may consider personalized hyperparameter control. For example, the main concept map could be tuned to be more concise, with fewer details, for learners with less experience or of low literacy, and so as video recommendations.

Future work may also consider adding personalized recommendation into the system when richer usage data become available.

Pilot browsing records along with past recommendations would be a good resource to guide future learners’ personal learning path.

Users may also participate in the construction of concept map.

As another line of future work, we may consider to provide finer grained guidelines for video learning across distant topics. For example, for topics (as captured by keywords) that have dependency on other key topics that are remote from the the current topics, mechanisms are needed to capture such inter-topical dependencies and offer materials on these remote topics to learners for a complete learning experience.

Besides the navigation and personalization of ConceptGuide, we may also consider improving the video sources to overcome the possible biases underlying the videos. Now ConceptGuide re-ranks top 50 videos using the algorithms described earlier to reduce the influence inherited from YouTube’s original ranking algorithms, e.g., ranking differences due to users’ search histories. In the future, we may sample a larger pool of videos from YouTube, as well as combine videos searched from multiple platforms, such as Vimeo and Twitch, to further mitigate this issue.

8 CONCLUSION

In this research, we designed a video learning system, Concept- Guide, based on YouTube educational videos, which automatically analyzed the content of videos and the comments of their viewers.

(12)

The interface showed an overview of the learning topic and recommended learners efficient learning sequences. To examine the effects of the learning system, we conducted an experiment to compare users’ learning experiences and learning outcomes between ConceptGuide and original YouTube search. The results showed the system was helpful for learning experiences on different aspects.

Participants could easily get a clear overall view of the topic and discover different parts of the topic. The system also helped them learn more systematically and confidently, thus enhanced learning efficiency and motivation.

ACKNOWLEDGEMENT

This work was supported in part by the Ministry of Science and Technology of Taiwan under grant No. 109-2221-E-009-123-MY3 and 109-2221-E-009-119-MY3, and by UC Davis through Hao-Chuan Wang’s startup grant. We thank Yun-Rou Lin for making the demo video.

REFERENCES

[1] John Adcock, Matthew Cooper, Laurent Denoue, Hamed Pirsiavash, and Lawrence A. Rowe. 2010. TalkMiner: A Search Engine for Online Lecture Video.

In Proceedings of the 18th ACM International Conference on Multimedia (Firenze, Italy). 1507–1508.

[2] Hermann Astleitner and Detlev Leutner. 1995. Learning Strategies for Unstruc- tured Hypermedia - A Framework for Theory, Research, and Practice. Journal of Educational Computing Research13, 4 (1995), 387–400.

[3] Fatiha Bousbahi and Henda Chorfi. 2015. MOOC-Rec: A Case Based Recom- mender System for MOOCs. Procedia - Social and Behavioral Sciences 195 (2015), 1813 – 1822.

[4] Peter Brusilovsky. 1996. Methods and techniques of adaptive hypermedia. User Modeling and User-Adapted Interaction6, 2 (01 Jul 1996), 87–129.

[5] Y. Chtouki, H. Harroud, M. Khalidi, and S. Bennani. 2012. The impact of YouTube videos on the student’s learning. In International Conference on Information Technology Based Higher Education and Training. 1–4.

[6] Kuo-Kuang Chu, Chien-I Lee, and Rong-Shi Tsai. 2011. Ontology Technology to Assist Learners’ Navigation in the Concept Map Learning System. Expert Syst.

Appl.38, 9 (Sept. 2011), 11293–11299.

[7] Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. 2009.

Introduction to algorithms. MIT press.

[8] C. Ailie Fraser, Julia M. Markel, N. James Basa, Mira Dontcheva, and Scott Klem- mer. 2019. ReMap: Multimodal Help-Seeking. In The Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST

’19). 96–98.

[9] C. Ailie Fraser, Tricia J. Ngoon, Mira Dontcheva, and Scott Klemmer. 2019. Re- Play: Contextually Presenting Learning Videos Across Software Applications. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19).

[10] Thomas MJ Fruchterman and Edward M Reingold. 1991. Graph drawing by force- directed placement. Software: Practice and experience 21, 11 (1991), 1129–1164.

[11] Ankit Gandhi, Arijit Biswas, Kundan Shrivastava, Ranjeet Kumar, Sahil Loomba, and Om Deshmukh. 2016. Easy Navigation Through Instructional Videos Us- ing Automatically Generated Table of Content. In IUI ’16 Companion (Sonoma, California, USA). 92–96.

[12] Qin Gao, Dunxing Wang, and Fan Gao. 2015. Impact of Knowledge Represen- tations on Problem-Oriented Learning in Online Environments. International Journal of Human-Computer Interaction31, 12 (2015), 922–938.

[13] Richard H. Hall and Angela O’Donnell. 1996. Cognitive and Affective Outcomes of Learning from Knowledge Maps. Contemporary Educational Psychology 21, 1 (1996), 94 – 101.

[14] Christoph HÃűlscher and Gerhard Strube. 2000. Web search behavior of Internet experts and newbies. Computer Networks 33, 1 (2000), 337–346.

[15] Google Inc. 2019. Google Cloud Natural Language. https://cloud.google.com/

natural-language/

[16] Troy Jones and Kristen Cuthrell. 2011. YouTube: Educational Potentials and Pitfalls. Computers in the Schools 28, 1 (2011), 75–85.

[17] Juho Kim, Elena Leah Glassman, Andrés Monroy-Hernández, and Meredith Ringel Morris. 2015. RIMES: Embedding Interactive Multimedia Exercises in Lecture Videos. In CHI.

[18] Juho Kim, Philip J. Guo, Carrie J. Cai, Shang-Wen Li, Krzysztof Z. Gajos, and Robert C. Miller. 2014. Data-driven Interaction Techniques for Improving Navi- gation of Educational Videos. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology(Honolulu, Hawaii, USA). 563–572.

[19] Patricia G Lange. 2019. Informal Learning on YouTube. The International Ency- clopedia of Media Literacy(2019), 1–11.

[20] Jae Hwa Lee and Aviv Segev. 2012. Knowledge Maps for E-learning. Computers

& Education59, 2 (2012), 353–364.

[21] Ching Liu, Juho Kim, and Hao-Chuan Wang. 2018. ConceptScape: Collaborative Concept Mapping for Video Learning. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). Article 387, 12 pages.

[22] Steven Loria. 2019. TextBlob: Simplified Text Processing. https://textblob.

readthedocs.io/en/dev/

[23] Debabrata Mahapatra, Ragunathan Mariappan, Vaibhav Rajan, Kuldeep Yadav, Seby A., and Sudeshna Roy. 2018. VideoKen: Automatic Video Summarization and Course Curation to Support Learning. In Companion Proceedings of the The Web Conference 2018(Lyon, France) (WWW ’18). 239–242.

[24] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).

[25] Joseph D Novak. 1990. Concept mapping: A useful tool for science education.

Journal of research in science teaching27, 10 (1990), 937–949.

[26] Angela M. O’Donnell, Donald F. Dansereau, and Richard H. Hall. 2002. Knowledge Maps as Scaffolds for Cognitive Processing. Educational Psychology Review 14, 1 (01 Mar 2002), 71–86.

[27] Liangming Pan, Chengjiang Li, Juanzi Li, and Jie Tang. 2017. Prerequisite relation learning for concepts in moocs. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vol. 1. 1447–1456.

[28] Sadhana Puntambekar, Agnes Stylianou, and Roland Hübscher. 2003. Improving Navigation and Learning in Hypertext Environments with Navigable Concept Maps. Hum.-Comput. Interact. 18, 4 (Dec. 2003), 395–428.

[29] Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. 2010. Automatic Keyword Extraction from Individual Documents. Wiley-Blackwell, Chapter 1, 1–20.

[30] M. Schwab, H. Strobelt, J. Tompkin, C. Fredericks, C. Huff, D. Higgins, A. Strezh- nev, M. Komisarchik, G. King, and H. Pfister. 2017. booc.io: An Education System with Hierarchical Concept Maps and Dynamic Non-linear Learning Plans. IEEE Transactions on Visualization and Computer Graphics23, 1 (Jan 2017), 571–580.

[31] Ruey-Shiang Shaw. 2010. A study of learning performance of e-learning materials design with knowledge maps. Computers & Education 54, 1 (2010), 253 – 264.

[32] Chareen Snelson. 2018. The Benefits and Challenges of YouTube as an Educational Resource. The Routledge Companion to Media Education, Copyright, and Fair Use (2018).

[33] YouTube API Service Team. 2018. YouTube Data API. "https://developers.google.

com/youtube/v3/".

[34] Feng Wang, Xiaoyan Li, Wenqiang Lei, Chen Huang, Min Yin, and Ting-Chuen Pong. 2015. Constructing Learning Maps for Lecture Videos by Exploring Wikipedia Knowledge. In Advances in Multimedia Information Processing – PCM 2015, Yo-Sung Ho, Jitao Sang, Yong Man Ro, Junmo Kim, and Fei Wu (Eds.).

559–569.

[35] Shuting Wang and Lei Liu. 2016. Prerequisite Concept Maps Extraction for AutomaticAssessment. In Proceedings of the 25th International Conference Com- panion on World Wide Web(Montréal, Québec, Canada) (WWW ’16 Companion).

519–521.

[36] Kuldeep Yadav, Ankit Gandhi, Arijit Biswas, Kundan Shrivastava, Saurabh Sri- vastava, and Om Deshmukh. 2016. ViZig: Anchor Points Based Non-Linear Navigation and Summarization in Educational Videos. In Proceedings of the 21st International Conference on Intelligent User Interfaces(Sonoma, California, USA) (IUI ’16). 407–418.

[37] Kuldeep Yadav, Kundan Shrivastava, S. Mohana Prasad, Harish Arsikere, Sonal Patil, Ranjeet Kumar, and Om Deshmukh. 2015. Content-driven Multi-modal Techniques for Non-linear Video Navigation. In Proceedings of the 20th Interna- tional Conference on Intelligent User Interfaces(Atlanta, Georgia, USA) (IUI ’15).

333–344.

[38] youtube-dl developers. 2018. youtube-dl. https://rg3.github.io/youtube-dl/.

[39] Baoquan Zhao, Shujin Lin, Xiaonan Luo, Songhua Xu, and Ruomei Wang. 2017.

A Novel System for Visual Navigation of Educational Videos Using Multimodal Cues. In Proceedings of the 25th ACM International Conference on Multimedia (Mountain View, California, USA) (MM ’17). 1680–1688.

[40] Jian Zhao, Chidansh Bhatt, Matthew Cooper, and David A. Shamma. 2018. Flexible Learning with Semantic Visual Exploration and Sequence-Based Recommenda- tion of MOOC Videos. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada) (CHI ’18). Article 329, 13 pages.