CHAPTER 5. Evaluation
5.4 Experiment details for proposition 3
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
100
Concluding, the usability of our model must be evaluated by comparing the customer growth rate and the accumulation of growth rate of ability and marketing, if possible. When customer growth rate is lower or equal to the accumulation of growth rate of ability and marketing, which indicates that one of the factor that influences interaction quality: effectiveness, cost and scarcity is low, our proposed model will become inappropriate for SMEs to comply with. However, if the effectiveness, cost and scarcity of interaction quality is high, which will makes customer growth rate higher than the accumulation of growth rate of ability and marketing, our proposed model will be very useful to SMEs to improve their service innovation value.
5.4 Experiment details for proposition 3
5.4.1 Experiment Challenges and Design Principles
After simulation testing, although our model is not effective in every case, we still discovered some situation that our model is able to provide support for SMEs. By this discovery, we then wish to test our proposed information system effectiveness in aiding the using of our model. However, unlike the previous experiments, the experiments in evaluating our system effectiveness cannot use simulation technique for the reason that we cannot simulate the user behaviour of our system due to that every user will have different using behaviour and it is impossible to design a simulation process to capture them all.
In addition, the effectiveness of interviews might also be limited in consideration that we will provide each user some short cases as guidance. The case guidance might be considered as the final output (section 4.6) of the system and will easily be how the customer evaluates this system. However, the quality of each cases are different; also, even given the same cases, different SME user might have totally different feelings and understanding of each cases. Hence, user interviewing might not be an effective
‧
how to find a proper interviewee – a SME with service innovation experience and are willing to build new alliance or interaction with other SMEs based on the suggestions – will arouse again as other experiments.Considering the difficulty that might happens within different kinds of system’s evaluation, Henver et al. (2004) had given 5 main types of system evaluation method for different system’s situation(Table 5.5). Within the 5 types of method, the reason for why type 1 and 3 is not applicable is listed before, and type 2,4 is more like measuring the efficiency of the system and is not fitting to the experiment purposes.
Accordingly, in this experiment, instead of using simulation techniques (type 3) or interviewing (type 1), we decide to describe the system thoroughly by giving a using scenario to prove its usefulness (type 5). The scenario will leads through all the actions within system using process, and gives an easier to understanding to how may the system aids users while they uses the system.
Table 5.5 Types of system evaluation method for different situation 1. Observational Case Study: Study artefact in depth in business environment
Field Study: Monitor use of artefact in multiple projects 2. Analytical Static analysis: Examine structure of artefact for static qualities
Architecture analysis: Study fit of artefact into technical IS architecture
Optimization: Demonstrate inherent optimal properties of artefact of provide optimally bounds on artefact behaviour Dynamic analysis: Study artefact in use of dynamic qualities 3. Experimental Controlled Experiment: Study artefact in controlled
environment for qualities
Simulation: Execute artefact with artificial data
4. Testing Functional Testing: Execute artefact interfaces to discover failures and identity defects.
Structural Testing: Perform coverage testing of some metric in
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
102
the artefact implementation.
5. Descriptive Informed Argument: Use information from the knowledge base to build a convincing argument for the artefact’s utility
Scenarios: Construct detailed scenarios around the artefact to demonstrate its utility
5.4.2 Experiment Design Details
The scenario will be an imagination SME owner Peter, who owns a hostel at Pillow Mountain Leisure Agriculture Area who needs to improve its service value to survive.
In the region, there are many different kinds of SMEs, like other hostels, food stands, restaurants, farms, bakery, and car rental company etc.; also, there are some travel agencies that have relationships with this regions and will help promoting. However, Peter is not interacting well and frequently with them, because Peter was putting his focus on improving his hostel before. Recently, Peter understands that what he can achieve by his own strength is limited, so he wishes to try another approach;
nevertheless, he has no idea about other approaches that he might able to use, so he start using this system.
5.4.2.1 Data Collection Process
At the beginning, Peter registered the account of the system, and then started inputting some basic data of him and his hostel. After the registration, Peter was presented with an input screen of the system where he can input data and interaction condition of him and his fellow SMEs, some possible candidates for Peter are presented already on the screen based on the location data Peter inputted during registration (Figure 5.16, also for the Annotation Module). Peter considered to input 6 more SMEs data as the following table 5.6 into the system (table 5.6 also include the
‧
data of the 4 candidates the system recommended), and considered them as a specific type of entity according to the interactions between them as table 5.7.
Table 5.6 Other SMEs details
SME name Descriptions
Hostel A Competitor, targets the same customer segment, and is cheaper and better than Peter’s hostel
Hostel B Competitor, but targets an even cheaper customer segment Bakery A A small bakery where Peter will buy breakfast for his customers Car rental A A car rental SME nearby Peter’s hostel, Peter sometime drives his
customer to there to rent cars.
Restaurant A A restaurant nearby where Peter will order for his customers sometimes
Restaurant B A famous restaurant nearby, Peter will introduced his customer to this restaurant sometimes
Souvenir shop A A nearby shop where Peter’s customer will go sometimes Souvenir shop B A bigger souvenir shop at the centre of the region.
Farm A Peter will take his customer to this farm to let them experience farming life if the customer asked.
Travel Agency A
A smaller travel agency, they will contact Peter to arrange
accommodations if they are planning for any activity to that region sometimes
Travel Agency B
A bigger travel agency, usually find other more famous hostels than Peters
Table 5.7 Interaction Details of other SMEs
SME name Interaction pattern Reason
Hostel A Lateral, Lv1 No direct relationship
Hostel B Lateral, Lv1 No direct relationship.
Bakery A Supplier, Lv1 Supplies food for Peter.
Car rental A Supplier, Lv1 Supplies services for Peter’s customer.
Restaurant A Supplier, Lv1 Supplies food for Peter.
Restaurant B Lateral, Lv1 No direct relationship.
Souvenir shop A Lateral, Lv1 No direct relationship.
Souvenir shop B Customer, Lv1 Can help promoting Peter’s hostel.
Farm A Lateral, Lv1 No direct relationship.
Travel Agency A Customer, Lv2 Can help promoting Peter’s hostel, and have done many businesses before.
Travel Agency B Customer, Lv1 Can help promoting Peter’s hostel
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
104
Figure 5.16 Data Collection Module enter page
In this module, Peter could create new entity easily by pressing the “add new”
button on the left upper corner (Figure 5.17); also, the system provides Peter a light solution to insert what the type of the entity and their cooperation level by drag and drop the entity into different boxes (Figure 5.18). In addition, to tackle the remaining constructs – power dependence and trust, Peter could click the buttons – “Power”,
“Trust” on the entity to change the level of these constructs (Figure 5.19). Finally, when Peter is not sure about the current status of his interaction pattern with another entity, he could click the “display” button to call the questionnaire screen out to fill in the more accurate data (Figure 5.20). The results of Peter’s input could be found in Figure 5.21.
Figure 5.17 Adding new entity
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
105
Figure 5.18 Dragging and drop the entity to desired location
Figure 5.19 Changing the Trust construct of entity by clicking the Trust button
Figure 5.20 Call out the questionnaire input box to fill in the data more accurately
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
106
Figure 5.21 Results of Peter’s input
5.4.2.2 Pattern Improvement Process
After Peter finished the data input process, the system will automatically execute the pattern recognition process to identify Peter’s current interaction pattern (customer type: lv2, supplier type: lv1, lateral type: lv1) and current service innovation value (Level 2 now), and then provide the interface for Peter to select and decide how he might want to change his interaction pattern and what degree of service innovation level he pursues. In addition, Peter can choose to activate the system’s support during the selection, in case that he might be unfamiliar about how to choose a SME to improve interaction pattern with (Figure 5.22, 5.23).
Peter selected to improve his interaction pattern with Restaurant A first with the hope that he could improve the total living quality of his hostel by providing better food supplied by Restaurant A (Figure 5.24). Due to this selection, based on the service innovation value calculate function, his service innovation value could be improved to level 3 because of having better interaction pattern with a supplier entity and a customer entity. If Peter had activated system support, then the system will suggest Peter to improve pattern with customer entity and supplier entity, based on our interaction pattern adjustment model. (Figure 5.25)
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
107
Figure 5.22 Pattern Adjusting Module- without system support
Figure 5.23 Pattern Adjusting Module- with system support (the customer section was erased from user’s choice)
Figure 5.24 Pattern Adjusting Module- After selected a target SME
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
108
Figure 5.25 Pattern Adjusting Module- After selected a target SME, the system will change the candidates automatically
With the help of the system support, Peter chooses to improve his interaction pattern with Restaurant A and Travel Agency A, in consideration that the previous one might be able to provide him better accommodation quality, and the latter one is supposedly capable to promote his hostels features (Figure 5.26).
Figure 5.26 Pattern Adjusting Module- After reach to service innovation value 5
5.4.2.3 Case Selection Process
After Peter selects the interaction pattern adjustments, the system analyses his choice and provides Peter some successful interaction pattern adjustment cases as
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
109
guidance (Ch4.6) to help him adjust his interaction pattern to achieve higher service innovation value.
Each pattern improvement that Peter chooses will have several corresponding cases;
for example like Peter had choose to improve his interaction pattern with Restaurant A (supplier), 4 cases are within the system are for improving interaction pattern with this kind of entity, so the system will display all the 4 cases to Peter to choose based on Peter’s own judgement.
In addition, for situation like Peter’s who had chosen 3 pattern adjustment steps (Restaurant A from level 1 to level 2, Restaurant A from level 2 to level 3, Travel agency A from level 2 to level 3), the system will show 3 groups of interaction pattern adjustment case to Peter, each for one pattern adjustment step (Figure 5. 26).
To make the interface neat, we choose to make the case’s content invisible until the user decide to see it. Peter could see the content of each case by clicking the “Read Content” button, and the case content will then be displayed on the screen (Figure 5.27).After selecting one case from one group (Figure 5.28), that group will become invisible, and will appear again only when the user decides not to use this case and press the delete button. When all groups have been selected, (for Peter’s case, when all the three groups had been selected), the submit button will appear (Figure 5.29), and will lead the user to next page and combine all the cases together. The cases will be sorted base on its belonging pattern choosing order (Figure 5.30).
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
110
Figure 5.26 Case selection interface – entering page
Figure 5.27 Case selection interface - Case content
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
111
Figure 5.28 Case selection interface – after selected one case
Figure 5.29 Case selection interface – after all required cases are selected
Figure 5.30 Case selection interface – All selected cases will be displayed after submit
5.4.3 System Using Scenario Conclusion
The purpose of this system existence is for assisting users to implement and make
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
112
use of our theory and model through information technologies. By showing a detail using scenario by imaginary user- Peter, we can see the effectiveness of how the system may support user using our model. Beginning with the data collection process, during this process, we’ve provided a simple but clear user interface for users to input their interaction data with other SMEs, which enables us to collect user’s interaction condition quickly and accurately. Secondly, the pattern adjustment process which corresponds to our interaction pattern recognizing module and interaction pattern adjustment module, is carefully designed with java scripts and ajax techniques to provide user a fluent using experience; also, we had designed a supporting function within this process, hence users could consider thoroughly about their interaction pattern improvement strategy without influenced by their information techniques or unfamiliar with the system.
Last, we had collected many interaction adjustment related cases for the users as guidance, and had grouped them base on the type of entity they are facing with and the degree of adjustment. Accordingly, users shall be able to find proper case for them to learn from easily, and can follow the case guidance step by step because the cases had been arranged according to patterns adjustment selections.
Concluding, although we are not able to test our proposed system in field by SME users, through the scenario simulating process, we can see this system is designed well in assisting user in using our theory of interaction pattern adjustment for service innovation. From the beginning data input process, the interface was designed to be as simple as possible to avoid unnecessary distraction and misleading; the pattern adjustment process was designed carefully to help user in using the system and get feedback immediately; the case selection process corresponds to previous steps, select proper case and provide an user-friendly interface for users to choose proper case for guidance. By these system design techniques, users shall be able to use our model
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
113
with ease by the help of this artifact.
Last but not least, this system could be improved when more SME users had used it to look for service innovation chances; also, this system is related with other system and theory which were mentioned in CH3 and hence more user data and using data could be obtained from other related system.