In this work, we propose CNERvis, a visual analytic system that assists users in interpreting and diagnosing the Chinese WS-POS-NER pipeline. We provide a complete solution for experts to interpret and diagnose the Chinese NER system using a complicated WS-POS-NER pipeline. Our tool helps users figure out low confidence NER prediction, focus on the problematic sub-modules, interpret the behavior of deep learning models, and understand how a decision is made to analyze a NER system in depth. In the end, we provide case studies to demonstrate the effectiveness of using our tool.
Our system has received positive feedback from the expert and succeeded in meet-ing the expert’s requirements. Meanwhile, the domain expert also pointed out the weaknesses. Although the training data view can find the incorrect labels in the train-ing dataset, our system is difficult to improve the model’s accuracy directly. We hope that CNERVis can provide novel methods to verify models when users find potential mistakes.
In addition, the limitation of our work is the scalability of the content view. The content view shows all characters of an article to help users check the correctness of the NER prediction. However, when the number of characters of an article is large,
the current design would be difficult for users to find the critical instance in the huge number of characters. We would need to extend our designs to facilitate exploration.
In the future, we also plan to expand our system to NER tasks in different domains, such as biomedical science. In biomedical text extraction, NER plays a crucial role by extracting meaningful information from clinical notes. The different domains have different demands in visual analytics. Furthermore, we also would like to apply our tool to more Chinese NLP tasks.
Bibliography
[1] Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jef-frey Dean, Matthieu Devin, Sanjay Ghemawat, GeofJef-frey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation, pages 265–283, 2016.
[2] Jason PC Chiu and Eric Nichols. Named entity recognition with bidirectional lstm-cnns. Transactions of the Association for Computational Linguistics, 4:357–
370, 2016.
[3] Kyunghyun Cho, Bart Van Merri¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase repre-sentations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
[4] Jianjing Cui, Jun Long, Erxue Min, and Yugang Mao. Wedl-nids: improving network intrusion detection using word embedding-based deep learning method.
In International Conference on Modeling Decisions for Artificial Intelligence, pages 283–295. Springer, 2018.
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[6] Alex Endert, William Ribarsky, Cagatay Turkay, BL William Wong, Ian Nabney, I D´ıaz Blanco, and Fabrice Rossi. The state of the art in integrating machine learning into visual analytics. In Computer Graphics Forum, volume 36, pages 458–486. Wiley Online Library, 2017.
[7] Kawin Ethayarajh. How contextual are contextualized word representations?
comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv:1909.00512, 2019.
[8] Francesco Gargiulo, Stefano Silvestri, Mario Ciampi, and Giuseppe De Pietro.
Deep neural network for hierarchical extreme multi-label text classification. Ap-plied Soft Computing, 79:125–138, 2019.
[9] Felix A Gers, J¨urgen Schmidhuber, and Fred Cummins. Learning to forget:
Continual prediction with lstm. Neural computation, 12(10):2451–2471, 2000.
[10] Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. Context-dependent fine-grained entity type tagging. arXiv preprint arXiv:1412.1820, 2014.
[11] Miguel Grinberg. Flask web development: developing web applications with python. ” O’Reilly Media, Inc.”, 2018.
[12] Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. Ontonotes: the 90% solution. In Proceedings of the human lan-guage technology conference of the NAACL, Companion Volume: Short Papers, pages 57–60, 2006.
[13] YA Joarder, Kh Mustafizur Rahman, and Fabiha Faiz Mahi. Uplifted tissue characterization and classification of fatty liver disease from ultrasound images.
Advancement in Image Processing and Pattern Recognition, 3(3), 2020.
[14] Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078, 2015.
[15] Zeynep H Kilimci and Selim Akyokus. Deep learning-and word embedding-based heterogeneous classifier ensembles for text classification. Complexity, 2018, 2018.
[16] Guan Li, Junpeng Wang, Han-Wei Shen, Kaixin Chen, Guihua Shan, and Zhonghua Lu. Cnnpruner: Pruning convolutional neural networks with visual analytics. IEEE Transactions on Visualization and Computer Graphics, 2020.
[17] Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and under-standing neural models in nlp. arXiv preprint arXiv:1506.01066, 2015.
[18] Peng-Hsuan Li, Tsu-Jui Fu, and Wei-Yun Ma. Why attention? analyze bil-stm deficiency and its remedies in the case of ner. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8236–8244, 2020.
[19] Shixia Liu, Xiting Wang, Mengchen Liu, and Jun Zhu. Towards better analysis of machine learning models: A visual analytics perspective. Visual Informatics, 1(1):48–56, 2017.
[20] Leland McInnes, John Healy, and James Melville. Umap: Uniform mani-fold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.
[21] Yao Ming, Shaozu Cao, Ruixiang Zhang, Zhen Li, Yuanzhe Chen, Yangqiu Song, and Huamin Qu. Understanding hidden memories of recurrent neural networks.
In 2017 IEEE Conference on Visual Analytics Science and Technology (VAST), pages 13–24. IEEE, 2017.
[22] Thai-Hoang Pham and Phuong Le-Hong. End-to-end recurrent neural network models for vietnamese named entity recognition: Word-level vs. character-level.
In International Conference of the Pacific Association for Computational Lin-guistics, pages 219–232. Springer, 2017.
[23] Frederick Reiss, Hong Xu, Bryan Cutler, Karthik Muthuraman, and Zachary Eichenberger. Identifying incorrect labels in the conll-2003 corpus. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 215–
226, 2020.
[24] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948.
[25] Ben Shneiderman. The eyes have it: A task by data type taxonomy for infor-mation visualizations. In The craft of inforinfor-mation visualization, pages 364–371.
Elsevier, 2003.
[26] Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexander M Rush. S eq 2s eq-v is: A visual debug-ging tool for sequence-to-sequence models. IEEE transactions on visualization and computer graphics, 25(1):353–363, 2018.
[27] Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M Rush. Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE transactions on visualization and computer graphics, 24(1):667–676, 2017.
[28] Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. Crossweigh: Training named entity tagger from imperfect annotations.
arXiv preprint arXiv:1909.01441, 2019.
[29] Xue Xia, Thaddeus Roppel, John Y Hung, Jian Zhang, Senthilkumar CG Pe-riaswamy, and Justin Patton. Environmental complexity measurement using shannon entropy. In 2020 SoutheastCon, pages 1–6. IEEE, 2020.
[30] Vikas Yadav and Steven Bethard. A survey on recent advances in named entity recognition from deep learning models. arXiv preprint arXiv:1910.11470, 2019.
[31] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer, 2014.
[32] Yue Zhang and Jie Yang. Chinese ner using lattice lstm. arXiv preprint arXiv:1805.02023, 2018.
[33] Ying Zhao, Feng Luo, Minghui Chen, Yingchao Wang, Jiazhi Xia, Fangfang Zhou, Yunhai Wang, Yi Chen, and Wei Chen. Evaluating multi-dimensional vi-sualizations for understanding fuzzy clusters. IEEE transactions on visualization and computer graphics, 25(1):12–21, 2018.
[34] Yuying Zhu, Guoxin Wang, and B¨orje F Karlsson. Can-ner: Convolu-tional attention network for chinese named entity recognition. arXiv preprint arXiv:1904.02141, 2019.