量化分析

本實驗目的會與黃喬(2016)LDA 主題模型，以及 Auto Summarizer、Free Summarizer、Tools4noobs 之線上開放原始碼摘要法模型進行比較，透過客

Perplexity 提示詞 TF 正 規化 章節數提示詞次數 Perplexity 提示詞次數 Chapter 數提示詞

*TF *TF 次數*TF

TF 0.3282 0.3273 0.3253

TF-ISF 0.3267 0.3281 0.3256

TF-SF 0.3239 0.3252 0.3222

平均 F1 值

TF 0.3247

TF-ISF 0.3252

TF-SF 0.3195

備註：粗體表示為最高值 資料來源：黃喬(2016)

由不同摘要器所產生的摘要結果，並與投影片進行比對計算 ROUGE-1、

ROUGE-2、ROUGE-3，以探討摘要擷取效能。我們設定擷取摘要的句子數 上限比較不同摘要器的優劣，因線上摘要器 Free Summarizer 無法設定句子 上限，及 Auto Summarizer 內部設定 5 個句子產生 1 個句子、10 個句子產生 2 個句子的方法，無法保證能依照不同上限句子的規則來產生摘要，如加入 探討可能會有疑慮，因此兩者並未做進一步的探討。

透過表 4-17 結果顯示，TF-ISF 所擷取的字詞數為最高，LSTM 則為最 低，原因在於不同摘要器的斷句方法不同，句子所涵蓋的字詞數而有所差 異，造成擷取摘要的字詞數有一定的差距。此外，TF-ISF 摘要器設定句子 上限較少時，比起 LSTM 的 ROUGE 結果較佳，是因當限制越少摘要句子 數中具有越高總字詞數時，因尚未達到參考摘要的總字詞數，因此最後得 出的 ROUGE 表現必然會越好；但當擷取摘要的總字詞數超過參考摘要的 總字詞數，ROUGE 表現則會漸漸遞減。

本論文透過最高 ROUGE 值及摘要字詞數來比較不同摘要器的表現，

三種摘要器限制摘要句子數上限為 25 時，同時具有最高的 ROUGE 表現。

LSTM 只需擷取 450 個平均字詞數能達到最高 31.57%的 ROUGE-1、11.85%

的 ROUGE-2 表現，Tools4noobs 則擷取 528 個平均字詞數才能達到最高的 5.78 的 ROUGE-3 表現，且並未明顯優於 LSTM 的 ROUGE-3 表現。但不同

TF-ISF 30.64 11.40 5.24 751 Summarizer 的詳細結果如表 4-20 所示；線上摘要器 Free Summarizer 的詳細 結果如表 4-21 所示；線上摘要器 Tools4noobs 的詳細結果如表 4-22 所示。

表 4-18 LSTM 深度神經網路語言摘要模型之 ROUGE(%)結果

文字稿名稱 R-1 R-2 R-3

Algorithms Design and Analysis Part 1 29.26 9.36 2.82 Algorithms Design and Analysis Part 2 34.13 10.98 3.57 Audio Signal Processing for Music Applications 20.14 5.98 2.05 automata 51.14 17.08 5.23 Beginning Game Programming with C 27.45 8.29 2.91 Climate Change 29.02 7.37 2.55 Computational Neuroscience 34.34 12.83 5.67 Discrete Optimization 34.06 11.03 3.55 Dynamical Modeling Methods for Systems Biology 37.91 16.46 9.91 Edx Introduction to Computer Programming Part 1 42.06 18.89 10.33

Edx Introduction to Computer Science 20.12 3.45 0.97 Experimental Methods in Systems Biology 31.88 9.42 3.94 Foundations of Virtual Instruction 28.77 14.07 8.42 Introduction to Chemistry Reactions and Ratios 28.85 10.09 3.99 Introduction to Data Science 26.58 9.68 5.51 Journalism Skills for Engaged Citizens 34.10 15.31 9.23 LINCS Data Coordination and Integration Center 30.28 10.81 5.49 Machine Learning 32.12 10.88 3.53 Malicious Software and its Underground Economy

Two Sides to Every Story 32.12 14.14 8.51

Natural Language Processing 2013 26.60 8.57 3.24 Network Analysis in Systems Biology 32.73 12.27 5.86 Probabilistic Graphical Models 31.71 9.04 2.77 Surviving Disruptive Technologies 39.19 19.63 13.55 Take the Lead on Healthcare Quality Improvement 42.53 22.19 14.75 The Brain and Space 29.95 11.54 5.45

Algorithms Design and Analysis Part 1 27.88 9.26 2.80 Algorithms Design and Analysis Part 2 33.92 11.16 3.44 Audio Signal Processing for Music Applications 21.30 7.59 3.42 automata 48.36 15.75 4.84 Beginning Game Programming with C 25.61 7.51 3.02 Climate Change 29.26 8.69 3.12 Computational Neuroscience 31.50 10.13 3.84 Discrete Optimization 29.21 8.02 2.01 Dynamical Modeling Methods for Systems Biology 34.16 13.03 6.64 Edx Introduction to Computer Programming Part 1 38.82 16.19 8.24 Edx Introduction to Computer Science 16.79 2.16 0.71 Experimental Methods in Systems Biology 31.71 9.71 4.03 Foundations of Virtual Instruction 29.51 12.78 8.13 Introduction to Chemistry Reactions and Ratios 28.54 9.50 2.81 Introduction to Data Science 25.20 9.25 5.37 Journalism Skills for Engaged Citizens 31.12 12.16 6.69 LINCS Data Coordination and Integration Center 32.83 12.58 6.49 Machine Learning 27.99 7.22 1.60 Malicious Software and its Underground Economy

Two Sides to Every Story 31.64 11.79 6.91 Natural Language Processing 2013 25.77 7.93 2.69 Network Analysis in Systems Biology 31.13 10.80 4.43 Probabilistic Graphical Models 28.48 7.29 1.96 Surviving Disruptive Technologies 36.56 18.02 12.31 Take the Lead on Healthcare Quality Improvement 39.43 18.67 11.37 The Brain and Space 28.84 9.64 4.11 Virology I How Viruses Work 25.66 6.16 1.57 Writing in the Sciences 26.59 9.65 5.21

平均 30.29 9.26 2.80 備註：R-1：ROUGE-1；R-2：ROUGE-2；R-3：ROUGE-3

表 4-20 Auto Summarizer 摘要模型之 ROUGE(%)結果

文字稿名稱 R-1 R-2 R-3

Algorithms Design and Analysis Part 1 26.83 9.04 2.61 Algorithms Design and Analysis Part 2 29.39 10.10 3.03 Audio Signal Processing for Music Applications 21.97 8.43 3.34 automata 18.80 7.71 2.94 Beginning Game Programming with C 24.13 7.65 2.55 Climate Change 25.13 8.03 3.36 Computational Neuroscience 19.84 6.37 2.17 Discrete Optimization 21.80 6.83 1.98 Dynamical Modeling Methods for Systems Biology 22.22 7.68 3.31 Edx Introduction to Computer Programming Part 1 27.27 10.81 4.79 Edx Introduction to Computer Science 10.88 1.89 0.64 Experimental Methods in Systems Biology 17.90 5.15 1.55 Foundations of Virtual Instruction 30.42 15.10 8.33 Introduction to Chemistry Reactions and Ratios 25.07 8.18 2.60 Introduction to Data Science 21.44 8.17 4.60 Journalism Skills for Engaged Citizens 15.66 5.98 3.06 LINCS Data Coordination and Integration Center 18.36 7.52 3.57 Machine Learning 21.77 5.16 1.31 Malicious Software and its Underground Economy

Two Sides to Every Story 21.84 8.17 4.50 Natural Language Processing 2013 22.67 7.72 3.04 Network Analysis in Systems Biology 20.30 7.32 3.28 Probabilistic Graphical Models 18.94 5.65 1.55 Surviving Disruptive Technologies 25.87 11.92 7.64 Take the Lead on Healthcare Quality Improvement 22.43 11.65 7.17 The Brain and Space 25.16 8.70 3.30 Virology I How Viruses Work 20.32 4.78 1.04 Writing in the Sciences 20.00 9.20 5.49

平均 22.09 7.96 3.44

備註：R-1：ROUGE-1；R-2：ROUGE-2；R-3：ROUGE-3 表 4-21 Free Summarizer 摘要模型之 ROUGE(%)結果

文字稿名稱 R-1 R-2 R-3

Algorithms Design and Analysis Part 1 29.50 10.08 2.96 Algorithms Design and Analysis Part 2 33.53 11.29 3.34

Audio Signal Processing for Music Applications 25.24 10.37 3.80 automata 28.55 11.17 3.88 Beginning Game Programming with C 27.94 8.52 3.12 Climate Change 29.23 8.37 2.98 Computational Neuroscience 28.23 9.49 3.60 Discrete Optimization 26.39 7.67 2.22 Dynamical Modeling Methods for Systems Biology 28.41 10.39 4.45 Edx Introduction to Computer Programming Part 1 35.74 14.42 6.67 Edx Introduction to Computer Science 16.14 3.04 1.43 Experimental Methods in Systems Biology 24.28 7.09 2.20 Foundations of Virtual Instruction 29.65 13.59 8.06 Introduction to Chemistry Reactions and Ratios 28.81 9.73 3.63 Introduction to Data Science 25.63 9.37 5.10 Journalism Skills for Engaged Citizens 23.57 10.56 5.99 LINCS Data Coordination and Integration Center 25.66 10.71 5.39 Machine Learning 23.81 5.92 1.99 Malicious Software and its Underground Economy

Two Sides to Every Story 27.76 11.62 7.01 Natural Language Processing 2013 25.13 7.84 2.81 Network Analysis in Systems Biology 26.28 9.13 3.82 Probabilistic Graphical Models 23.37 6.22 1.47 Surviving Disruptive Technologies 33.94 15.84 10.24 Take the Lead on Healthcare Quality Improvement 31.68 16.70 10.59 The Brain and Space 28.41 10.45 4.47 Virology I How Viruses Work 24.59 5.77 1.36 Writing in the Sciences 24.58 10.08 5.50

平均 27.26 9.83 4.37

備註：R-1：ROUGE-1；R-2：ROUGE-2；R-3：ROUGE-3 表 4-22 Tools4noobs 摘要模型之 ROUGE(%)結果

文字稿名稱 R-1 R-2 R-3

Algorithms Design and Analysis Part 1 24.77 7.95 2.22 Algorithms Design and Analysis Part 2 31.03 9.47 2.74 Audio Signal Processing for Music Applications 17.88 5.47 1.93 automata 39.03 11.84 3.11 Beginning Game Programming with C 21.09 5.41 1.64 Climate Change 27.30 7.81 3.06 Computational Neuroscience 26.53 7.97 3.56 Discrete Optimization 22.08 5.68 1.42 Dynamical Modeling Methods for Systems Biology 31.43 10.97 5.28 Edx Introduction to Computer Programming Part 1 33.31 13.15 6.42

Edx Introduction to Computer Science 16.31 1.40 0.15 Experimental Methods in Systems Biology 27.96 7.70 2.27 Foundations of Virtual Instruction 23.42 9.29 5.31 Introduction to Chemistry Reactions and Ratios 25.34 7.73 2.31 Introduction to Data Science 21.54 6.71 3.45 Journalism Skills for Engaged Citizens 20.57 6.28 2.97 LINCS Data Coordination and Integration Center 24.93 9.70 4.99 Machine Learning 24.99 6.25 1.53 Malicious Software and its Underground Economy

Two Sides to Every Story 23.97 7.69 4.09 Natural Language Processing 2013 23.48 7.11 2.78 Network Analysis in Systems Biology 27.19 9.57 4.09 Probabilistic Graphical Models 27.54 6.59 1.75 Surviving Disruptive Technologies 28.37 11.65 7.31 Take the Lead on Healthcare Quality Improvement 32.56 14.69 8.56 The Brain and Space 24.26 7.92 2.67

圖 4-2 不同摘要器方法之平均 ROUGE 結果比較

透過 ANOVA 之 P 值，能了解本論文所採用的 LSTM 深度神經網路語 言摘要模型與其他模型方法模型之間是否有顯著差異，由表 4-23 結果顯示，

ROUGE-1、ROUGE-2、ROUGE-3 之 P 值都小於 0.05，代表不同模型之間 的摘要方法具有顯著差異。

表 4-23 摘要器方法間的 ANOVA 分析 P 值 ROUGE-1 < 0.0001***

ROUGE-2 < 0.0001***

ROUGE-3 0.0017***

備註：*p<0.05；**p<0.01；***p<0.001 0%

5%

10%

15%

20%

25%

30%

35%

ROUGE-1 ROUGE-2 ROUGE-3

LSTM TF-ISF

Auto Summarizer Free Summarizer Tools4noobs

第伍章 結論

線上課程的發展，對學習者而言是有助於學習的平台。以往學習者想 了解其內容，需看過大部分的內容才能了解課程想表達的意涵，如能提供 一套協助學習者快速掌握課程內容，應能輔助學習者更節省時間的花費。