第五章 系 使用分析 統
5.3 相 程度 關
5.3.3 目完成率 題
如表 24，本研究發現只有平均 生發 編譯 期時 錯誤 數與題次 目完成率呈高 著地中度 相顯 負 關(Max. |r|=0.52; p<0.01)，如 圖 41。依實際觀察，班上 生通常是在學 準備好要 交作繳 業時才使 用OJ，因 在為 OJ 撰 比不上在碼 IDE 舒 ，所以在適 OJ發生的編譯時 錯誤期 都只是從 IDE 複製 原始碼時忘了改類別 稱名 或是落掉下括弧。 理 是所有 生都能快速自行解 的 小這 應 學 決 極 問題，
目完成率呈中度相 ， 部份可能 生使用 工具的能力有 ，而 程式基本概
卻與題 關 這個 與學 資訊 關 與
念無關，是否能在早期透過觀察所有 生使用 程工具的 象學 課 現 預測往後需要 助的族群是更協 進 一步的議題。
表 24. 目完成率相題 關係數階 統計段 表
Milestone Avg. CE
Quiz 1 r=-0.52; p<0.01 Midterm Exam r=-0.39; p<0.01 Quiz 2 r=-0.35; p=0.01 Quiz 3 r=-0.48; p<0.01 Final Exam r=-0.42; p<0.01
圖 41. 目完成率因子題
doi:10.6342/NTU201701661
Milestone Finish Rate (%) Avg. TLE Avg. CE Quiz 1 r=0.43; p<0.01 r=-0.38; p=0.01
Midterm Exam r=0.68; p<0.01 r=-0.39; p<0.01 Quiz 2 r=0.58; p<0.01
Quiz 3 r=0.39; p=0.01 Final Exam r=0.46; p<0.01 Semester Grade r=0.72; p<0.01
86
圖 42. 第一次小考分 因子數 圖 43. 期中考分 因子數
圖 44. 考 分 因子試 數
doi:10.6342/NTU201701661
5.4 預測 模型
本研究以Therneau 與 Atkinson (2015) 作的實 R語言CART(Classification And Regression Tree)(Breiman et al., 1984)套件 rpart 去訓練表 18 的所有因子，在每個節點至少要有一筆資料 的 件下得到 度 合條 過 擬 (Over Fitting)的第一次小考等第分類樹(Acc.=100%)如 圖 45。於是本研 究 再 使 用 10 折 交 叉 驗證 (10-Fold Cross Validation) 得 到 10 組誤差 率 ， 最 後 挑選最 佳 的 CP(Complexity Parameter)使得 差率平均值誤 (Cross-Validation Error) 最小如 為 圖 46。 最佳的將 CP(cp=0.14)重新 入帶 rpart 函式演算，得到修剪 的第一次小考等第分過 類樹(Acc.=51.92%)如 圖 47。
88
圖 45. 第一次小考等第分類樹
圖 47. 第一次小考等第分 剪枝類 樹 圖 46. CART複雜 參數對 圖度 照
doi:10.6342/NTU201701661
5.5 用性 適 討論
本研究發現預測準確 (Acc.=51.92%)相 低，而且分率 當 類樹只把75% 目完成率 成分題 當 類學 生第一次小考等第的閥值。回顧前人所提出的 Error Quotient (Jadud, 2006)、Watwin Score (Watson et al., 2013)主要都是統計學生在IDE 撰 的碼 編譯 數次 及所花 的費 時間 採 這 與， 用 些 特
第六章 結論
doi:10.6342/NTU201701661
6.3 未 工作 來
doi:10.6342/NTU201701661 1-2) 模擬終端： 然雖 說當批改 本 行 生程式 ，一次性地接收由腳 執 學 時 標準輸 給入 定的測試
料是最 便的作法，可是 生
資 為簡 當學 從 Eclipse 提供的 Console 批次地練習 鍵盤輸 測試資由 入 料 習慣後， 於對 TAFree 一次性的 入輸 與輸 會出 有不適應 問題的 。未 可以仿效來 Wilcox (2016) 控 制 生程式的 入學 輸 與輸出串流，偵測執 緒行 等待 入的輸 狀態，再 予給 測試資料，回傳對應 輸的 出， 且判 是否超並 斷 過執 時間行 限制，以 到模仿 端的效果。達 終
1-3) 部份 分：教 人 可以在批改 器人上安給 學 員 機 裝 JUnit 框架以利批改 本腳 對執 階 僅行 段 有文字 出的 目做 元輸 題 單 測試。透過單 測試元 ，教 人 可以學 員 對測試程式中的所有方法配分，
一 方法只能包含一個 個測試目的，最後累 通計 過測試方法的分 作數 為該題的得分，部份 分給 機 制將對 統應系 用於 限 短的考時 較 試現場 幫有 助，但前提是， 生程式要能學 編譯成功。
94
參 考文 獻
Ahadi, A., R. Lister, H. Haapala, & A. Vihavainen (2015). “Exploring Machine Learning Methods to Automatically Identify Students in Need of Assistance.” In Proceedings of the 11th annual International Conference on International Computing Education Research (ICER '15), Omaha, Nebraska, USA, 121-130.
Ahadi, A., R. Lister, & A. Vihavainen (2016). “On the Number of Attempts Students Made on Some Online Programming Exercises During Semester and their Subsequent Performance on Final Exam Questions.” In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '16), Arequipa, Peru, 218-223.
Aiken, A. Moss [Online]. Available from http://theory.stanford.edu/~aiken/moss/
Ala-Mutka, K. M. (2005). “A Survey of Automated Assessment Approaches for Programming Assignments.“ Computer Science Education, 15(2), 83-102.
Albuquerque, D., B. Cafeo, A. Garcia, S. Barbosa, S. Abrahão, & A. Ribeiro (2015), “Quantifying Usability of Domain-Specific Languages: an Empirical Study on Software Maintenance.”
Journal of Systems and Software, 101, 245-259.
Association of Computing Machinery. ACM-ICPC [Online]. Available from http://icpc.baylor.edu/
Bergin, S. & R. Reilly (2005). “Programming: Factors that Influence Success.” In Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '05), St.
Louis, Missouri, USA, 411-415.
Bishop, J. & M. A. Verleger (2013). ”The Flipped Classroom: A Survey of the Research.” In Proceedings of the 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia, 23.1200.1-23.1200.18.
Booch, G., J. Rumbaugh, & I. Jacobson (2005). The Unified Modeling Language User Guide (2nd ed.). Addison-Wesley Professional.
Breiman, L., J. Friedman, R. A. Olshen, & C. J. Stone (1984). Classification and Regression Trees. Belmont, CA: Wadsworth International Group.
doi:10.6342/NTU201701661 Constable, R. L. (1980). “The Role of Finite Automata in the Development of Modern Computing
Theory.” In Proceedings of The Kleene Symposium, North-Holland Publishing Co., 61-83.
Du, X., C. Yi, Y. Wei, S. Feng, & Z. Gong (2010). “Design of Automata Online Judge.” In Proceedings of the 2nd International Conference on Information Engineering and Computer Science (ICIECS), Wuhan, China, 1-4.
Forišek, M. (2006). “Security of Programming Contest Systems.” In proceedings of the 2nd International Conference on Informatics in Secondary Schools: Evolution and Perspectives (ISSEP), Vilnius, Lietuvos, 553-563.
Fowler, M. (2010). Domain-Specific Languages (1st ed.). Addison-Wesley Professional.
Friedl, J. E. F. (2006). Mastering Regular Expressions (3rd ed.). O'Reilly Media, Inc.
Google Inc. Google Code Jam [Online]. Available from http://code.google.com/codejam/
Gronback, R. C. (2009). Eclipse Modeling Project: A Domain-Specific Language (DSL) Toolkit.
Addison-Wesley Professional.
Hanks, B. (2008). “Problems Encountered by Novice Pair Programmers.” Journal on Educational Resources in Computing (JERIC), 7(4), 1-13.
Ihantola, P., T. Ahoniemi, V. Karavirta, & O. Seppälä (2010). “Review of Recent Systems for Automatic Assessment of Programming Assignments.” In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling '10), Koli, Finland, 86-93.
Insa, D. & J. Silva (2015). “Semi-Automatic Assessment of Unrestrained Java Code: a Library, a DSL, and a Workbench to Assess Exams and Exercises.” In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '15), Vilnius, Lithuania, 39-44.
Jadud, M. C. (2006). “Methods and Tools for Exploring Novice Compilation Behaviour.” In Proceedings of the 2nd International Workshop on Computing Education Research (ICER '06), Canterbury, United Kingdom, 73-84.
JUnit. JUnit 5 [Open source project]. Available from http://github.com/junit-team/junit5
96
Kaohsiung Affilicated Senior High School. ZeroJudge [Online]. Available from http://judge.nccucs.org
Kleene, S. C. (1956). “Representation of Events in Nerve Nets and Finite Automata.” In C. E.
Shannon & J. McCarthy (Eds.), Automata Studies (pp. 3-41). Princeton, NJ: Princeton University Press.
Kurnia, A., A. Lim, & B. Cheang (2001). “Online Judge.” Computers & Education, 36(4), 299-315.
Liang, Y. D. (2015). Introduction to Java Programming, Comprehensive Version (10th ed.). Prentice Hall.
McCulloch, W. S. & W. Pitts (1943). “A Logical Calculus of the Ideas Immanent in Nervous Activity.” Bulletin of Mathematical Biophysics, 5(4), 115-133.
Mizuno, T. & S. Y. Nishizaki (2014). “Distributed Online Judge System for Interactive Theorem Provers.” EPJ Web of Conferences, 68, 00016.
Pettit, R. S., J. D. Homer, K. M. Holcomb, N. Simone, & S. A. Mengel (2015). “Are Automated Assessment Tools Helpful in Programming Courses?” In Proceedings of the 2015 ASEE Annual Conference & Exposition, Seattle, Washington, 26.230.1-26.230.20.
Resnick, M., J. Maloney, A. Monroy-Hernández, N. Rusk, E. Eastmond, K. Brennan, A. Millner, E.
Rosenbaum, J. Silver, B. Silverman, & Y. Kafai (2009). “Scratch: Programming for All.”
Communications of the ACM, 52(11), 60-67.
Robins, A., P. Haden, & S. Garner (2006). “Problem Distributions in a CS1 Course.” In Proceedings of the 8th Australasian Conference on Computing Education - Volume 52 (ACE '06), Australian Computer Society, Inc., Darlinghurst, Australia, Australia, 165-173.
Sams, A. & J. Bergmann (2013). “Flip Your Students' Learning.” Educational Leadership, 70(6), 16-20.
Sanders, W. (2013). “Using UMLs with Design Patterns.” In PHP Design Patterns (pp.59-78).
Sebastopol, CA: O'Reilly Media, Inc.
doi:10.6342/NTU201701661 Solms, F. & V. Pieterse (2016). “Towards a Generic DSL for Automated Marking Systems.” In
Proceedings of the 45th annual Conference of the Southern African Computer Lecturers' Association (SACLA 2016), Springer International Publishing, 59-66.
Suhailan, S., A. S. Shibghatullah, & M. A. Burhanuddin (2014). “Perspective of Automated Programming Error Feedback Approaches in Problem Solving Exercises.” Journal of Theoretical and Applied Information Technology, 70(1), 121-129.
TestFX. TestFX [Open source project]. Available from http://github.com/TestFX/TestFX
Therneau, T. M. & E. J. Atkinson (2015). An Introduction to Recursive Partitioning Using the RPART Routines. Mayo Foundation.
Thompson, K. (1968). “Programming Techniques: Regular expression search algorithm.”
Communications of the ACM, 11(6), 419-422.
Tremblay, G., F. Guérin, A. Pons, & A. Salah (2008). “Oto, a Generic and Extensible Tool for Marking Programming Assignments.” Software: Practice and Experience, 38(3), 307–333.
Tremblay, G. & P. Lessard (2011). “A Marking Language for the Oto Assignment Marking Tool.” In Proceedings of the 16th annual joint Conference on Innovation and Technology in Computer Science Education (ITiCSE '11), Darmstadt, Germany, 148-152.
University of Valladolid. UVa Online Judge [Online]. Available from http://uva.onlinejudge.org Watson, C., F. W. B. Li, & J. L. Godwin (2013). “Predicting Performance in an Introductory
Programming Course by Logging and Analyzing Student Programming Behavior.” In 2013 IEEE 13th International Conference on Advanced Learning Technologies, Beijing, China, 319-323.
Wilcox, C. (2016). “Testing Strategies for the Automated Grading of Student Programs.” In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (SIGCSE '16), Memphis, Tennessee, USA, 437-442.
98
附錄 原始碼專案網址對照表
章次 專案名稱 專案描述 專 網案 址
致謝 TAFreeUI Some front-end drafts for TAFree1.0 https://github.com/TAFree/TAFreeUI
第三章 TAFree1.0 The first generation of online judge for NTU CE Computer Programming course.
https://github.com/TAFree/TAFree1.0
第三章 Ghassho A judge client for TAFree1.0
executing in Docker container. https://github.com/TAFree/Ghassho
第四章 JudgeBoy A PHP repository offering examples for customizing judge script. https://github.com/TAFree/JudgeBoy