Fan Wei Min
Doctoral student, Dept. of Library and Information Science National Taiwan University, Taiwan (R.O.C.)
E-mail: D05126005@ntu.edu.tw
Keywords: Chatbot; Library Reference Service; User Experience; Virtual Consult Reference
【Abstract】
Many industries have implemented chatbot application for online custom service.
Considering the increasing reference consults and the declining manpower, the university library could take chatbot application as a solution. This study focus on the implemetation and evaluation of chatbot on virtual reference service of university libraries. We inducted the virtual reference literature for the dialouge pattern and question types and choose 78 articles of university library reference service blog to generate 86 intents(dialouge pattern) and 93 entities (134 synonyms) for virtual reference service chatbot. After the implementation, 32 users with the background of the library information science were invited for testing. As the result, the most interactive dialogue topic were “my search for books”, “opening hours”, “pre‐set greetings”,
“study room seats” and the other library resources. The statistics showed that the scores of information quality and interactivity were significant improvements, while the system quality score was almost significant. In terms of perceptual usefulness and perceived ease of use, two aspects were significantly improved. Users thought virtual reference service chatbot was easy to get starte. It could answer with good quality and could solve to users’ needs. Addiotnally, it could save the search time for the user or reduce the access steps. However, the avergage score of user’s attitude was improved but not significant, and the avergage score of user’s intention was significant lower. The score of Recommendation willingness was low. According to the user feedback, they believed the performance of chatbot. However, they did have more expectation in converating more influently to the chatbot and asking more questions beyoud the reference question template. Overall, the chatrobot application to virtual reference services is extremely
DOI: 10.6245/JLIS.202004_46(1).0001
feasible. University libraries can use chatbot application to provide virtual reference services and considerate to apply the technology of natural language process in order to solve the information need and raise the satisfaction from users.
【Long Abstract】
Introduction
In recent years, the demand of virtual reference service in libraries of universities has been increased in contrast to physical reference service in libraries. However, the human resource to support reference service has been decreased year by year and the problem has been an urgent issue for libraries to solve.
As chatbot service technology has developed, many industries start to use chatbot technology to cover the human resource gap such as the reservatio of taxi, the table reservation of restaurants or the promoting of tea drinks. The implement of chabot technology on virtual reference service may be a solution for providing some support for the shortage of human resource and also brings a lot of advantages. For readers, it is much easier to submitting questions to chatbot service through mobile device rather than browsing web pages by themselves, and this way can reduce readers’ anxiety or embarassment without face-to-face consultation. For librarians, the chatbots can provide 7/24 service hours, one-to-many instant service, the consistent answer template. The chatbot can answer the easy questions and leave the really personal or complex questions to librarians. Furthermore, chatbots can easily embedded in any platforms to guide readers to fimilar with library service and help librarian to analyze the service satisfication through transaction logs to re-evaluate the service (Vincze, 2017). The library can make the collection deveoplemt according to the log records. This study tried to build a chatbot service of virtual reference service and evaluate usability of the chatbot service. Hope this study experience could be a reference for the other librarians.
Method
The study has two phases. Built the chatbot service for virtual reference service in first phase and did the usability evaluation of chatbot service. In first phase, the researcher studied the literature review to integrate the reference questions type and design the dialouge pattern. The reference question type included five types: (1) ready reference; (2) subject-based research question; (3) policy and procedural question; (4) directional questions; (5) others or referrals (Numminen & Vakkari, 2009). The subject-based research questions need the customized reference service so this type refernece question was excluded from the chatbot service. The dialogue pattern included five steps: (1) greeting and acknowledgement; (2) two to four sources or answers depending on the type of question; (3) name or
citation for the sources; (4) search descriptions; (5) closing (Burger et al., 2010). This study chose an university library blog because it is an authorized reference service blog in Taiwan and it has been set up since 2010 and maintained by the professional subject librarians. The chatbot service used Google DialogFlow to design the conversation template and was implemented in Facebook messenger. In order to evaluate the usability, this study invited the subjects with Library Information Science (LIS) background do the usability test in second phase. Before and after the experiment, the subjects were required to fulfill the questionnaire. Xu (2016) adopted the Technology Acceptance Model (TAM) and designed the questionnaire with seven variables including (1) Information Quality; (2) System Quality;
(3) Interactivity; (4) Perceived Usefulness; (5) Perceived Use-of-Ease; (6) User Intention; (7) User attitude. This study used SPSS software to do the Paired Sample t test and summarized the answers of open questions.
Results
There was total 78 reference blogs articles selected as knowledge base of the chatbot service and the reference question type including (1) ready reference (77%); (2) policy and procedural question (12%) ; (3) directional questions (6%) ; (4) others or referrals (5%). It further generated 86 intents (dialouge pattern) and 93 entities (134 synonyms) for virtual reference service chatbot in Google DialogFlow. The total number of subjects was 32 (8 males and 24 females) and they were librians or graduate students of Library Information Science. Twelve subjects were librarians and also graduated student. The most interactive dialogue topic were “my search for books”, “opening hours”, “pre-set greetings”, “study room seats” and the other requests for library resources. The average score of pretest and post-test were 4.49 and 4.90 and it was raised 0.41 points. The results of the Paired Sample t test were described as follows.
(1) Information Quality t(31)=2.80 , p=0.009 and Interactivity t(31)=3.91 , p=0.000, users highly affirmed the Information Quality and Interactivity of the chatbot service, and they were satisfied with the way and the content of chatbot answering.
(2) System Quality t(31)=1.99 , p=0.055, the average score increased by 0.42 points, t(31)=1.99, p=0.055 was close to significant and users had positive comments of the quality of the chatbot system.
(3) Perceived Usefulness t(31)=4.22, p=0.000 and Perceived Ease-of-use t(31)=3.03, p=0.005, users were highly affirmative of the usefulness and ease of use of chatbots. It showed that users thought the chatbot was convenient to use and easy to use, and could save users the steps as well as time for retrieving.
(4) The average score of User Attitude increased by 0.10 points, and t(31)=0.558, p=0.588. Although the user score was slightly improved, but not significant, the User Intention-oriented score was a significant decrease of 0.74 points (t (31)=-4.53, p=0.000).
Variable Std. Deviation Std Error Mean t df Sig. (2-tailed) Information Quality 1.22 0.22 2.80 31.00 0.009***
System Quality 1.20 0.21 1.99 31.00 0.055***
Interactivity 1.23 0.22 3.91 31.00 0.000***
Perceived Usefulness 1.28 0.23 4.22 31.00 0.000***
Perceived Ease-of-Use 1.29 0.23 3.03 31.00 0.005***
User Intention 1.08 0.19 0.55 31.00 0.588***
User Attitude 0.92 0.16 -4.53 31.00 0.000***
This study was to further explain the relations between variables and the statistical samples considered were not normally distributed. Therefore, we used Spearman correlation to analyze them and results were described as follows. Interacitivity, Information Quality, and System Quality had a significant impact on Perceived Usefulness, Perceived Ease-of-Use and user attitude. However, only System Quality didn’t have impact on the User Intention. After using the chatbot service, users gave positive their attitude toward the service, but they didn’t have motive to use it because of the System Quality. Due to the open questions, the reason was that users expected that the answering scope of reference questions is not limited to Q&A , and they would like to use natural language to chat with service robots.
0.553**
0.481** 0.482**
0.357*
0.628**
0.569**
0.447**
0.227
0.461** 0.474**
0.638**
0.577**
PERCEIVED USEFULNESS PERCEIVED EASE-OF-USE USER ATTITUDE USER INTENTION Information Quality System Quality Interactivity
Conclusion
The professional users had good comments on the chatbot service of virtual reference service including Information Quality, Interactivity, Perceived Usefulness, and Perceived Ease-of-Use. Although users had much more expectation on using natural language to interact with the chatbot beyond the specific topics, they still gave the chatbot service positive attitude. We can say that it is highly feasible to implement chatbot technology in virtual reference service. The natural processing technology in Chinese module in Google DialogFlow is continuing developing. Universities libraies need to develop tools to automatically abstarct the blogs and transform the blog content into the dialogue template. It may be possible to cover more information like the FAQ or the reference books of each course. It also can push more important service notice during the exam week or graduation period. In addition, the subjects in this study were limited to those with LIS background. The subjects can be further extended to the common users in order to understand their deman and behavioal pattern. We expect the opinions from different discipline can help to optimize the chatbot service of virtual reference service.
【Romanization of references is offered in the paper.】