CHPATER 6 EVALUATION
6.2 A SSUMPTIONS AND E XPERIMENT D ESIGN D ETAILS
6.2.2 Experiment Design Details
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
98
Assumption 2: On the other hand, the aim of the research evaluation is to examine the effectiveness of the architecture which we mentioned above, based on the Maslow theory and Dramatica to construct to moving story structure to incite SMEs to do service innovation. However, the reading inclination of the testers will be influenced by the writing content. In other words, it’s not only to the story structure but also the writing content’s problem that would affect the testers whether to want the continued reading.
Assumption 3: As we mentioned in the previous chapter, if want to convince other to do innovation; the homogeneity is an important element (Rogers, 1962).
On the other hand, Rogers also suggested that if we want to convince other to do service innovation, we should start from the past experiences. Based on the above theory, the most outstanding feature of the story automatic generator proposed by this research is it can utilize personal information to establish the story protagonist. For the automatic purpose we need to encode the user’s information; however, this will cause the customized not enough and the lower correlation. From another direction to think about this phenomenon, the position of this research’s mechanism is the advertisement to convince (not the advertisement to disseminate) the SME, therefore, the lower correlation is acceptable.
6.2.2 Experiment Design Details
Given that it’s important to test the incentive behavior before the system is put into practice. However, the incentive behavior involves with human psychological motivation factors which can’t measure by the computer program. Alternatively, we examine the effectiveness by country field investigation in the Mt. Pillow
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
99
Recreational Agriculture Area in Ilan County. For the purpose of testing the propositions, two different dimensions should be dilated: the system UI process and the experimental subject.
System UI Process
Figure 6.1 The system UI process and internal process
Figure 6.1 is the system UI process and internal process. Form the experimental subject’s point of view; there are several steps after they join the mini advertising story generator system. First, they need to administer the questionnaire and let the system to analyze their background information and choose the appropriate story framework and story plot for them. As we mentioned in the previous chapters, there are four story frameworks (Safety needs, Love needs, Esteem needs and Self-actualization needs) and the story plot divided into two types (Learning and Cooperation).
‧
After the system recommends the experimental subject the most suitable one, the experimental subject will have two choices, with the relevant background situation or not relevant. The previous selection will allow user to become the story protagonist, and the name, age, birthplace, etc. of the protagonist are as same as the experimental subject. The other select then let user read the other person’s story.
The last step presents the randomly generated story paragraph to the experimental subject, and based on their free will to select if they want to move on or not. On the other hand, there are several random paragraphs written by the advertising designer who works in the advertising industry in the internal processes, and utilize random manner to decide which paragraph will be showed.
The Experimental Subjects
In addition to the system UI process, the other significant test element is the experimental subjects. As mentioned in the previous chapters, this research takes the Mt. Pillow Recreational Agriculture Area in Ilan County as the study area. Mt. Pillow Recreational Agriculture Area is the first leisure agriculture area in Ilan which was established in 2000. The geography location includes four villages in which the Pillow Village is the main planning area, and it covers an area about 78 hectares. The distinguishing feature of the Mt. Pillow is the luxuriant natural resource which is the largest fruit-producing areas in Ilan, mainly in the fruiter cultivation and including more than 30 kinds of fruit that provide fruit picking service. Because the unique natural ecosystems and environmental resources making the Mt. Pillow Recreational Agriculture Area develop rapidly, embracing the various types of experience farm and the B & B restaurants with different themes can provide tourist with the experience the colorful rural life.
Owing to the operators of leisure agriculture being SMEs whose human resources and management experience are deficient and incomplete, they are bound to
‧
face many operational problems. This research chooses Mt. Pillow Recreational Agriculture Area in Ilan County as a case study area. Through interview and observations, this study aims to understand the SME operators of leisure agriculture to and their problems of the current status in the leisure agriculture business and regional development. The interview process by the dictation way and the interview content include two parts; demonstrating the system and collecting the opinion and reaction from the ownerships. Based on the opinions and reaction conveyed by the ownerships, we intend to seek out some evidence to confirm the truth to support our propositions.
The following table shows the details of interview which are the background of every experimental subject and the type of interview.
Table 6.1 The details background of every experimental subjects
Number Name Business Type Business Content
The business is Bed-and-Breakfast running. The selling points are the beautiful scenery and the night scenes. The other principal occupation is providing vegetables to convenience stores and McDonald's.
3 Subject 3
Bed-and-Breakfast Industry
The main business is Bed-and-Breakfast running, and have the garden café constructed based on the ownership’s experience, tourist teaching of fruit growth process and DIY branches insects teaching. planting, the tourists picking fruit experience, fruit ice cream DIY and fruit ice cream selling.
‧
Bed-and-Breakfast Industry. For the above reason, the principal business type of experimental subject which we chose focus on the Compound Experienced Leisure Farms and Bed-and-Breakfast Industry. In connection with these five experimental subjects we took the focus group interview and in-depth interview to comprehend them from various angles deeply. Table 6.2 shows the details of the interview process from these two perspectives: focus group interview and in-depth interview.Table 6.2 The type of interview
Definition Interview Situation their perceptions and opinions. The researchers introduce the subject of interview, and encourage the membership of small group participate in the discussion without the researchers themselves. The shortcoming of the focus-group interview is that the reaction of each membership is not independent and it’s easily affected by the dominant group member (Henderson & Naomi R, 2009).
We gathered three experimental subjects to do the focus-group interview. First, we illustrated the research objective and vision, and then demonstrated the system process and let them to go through the integral system process. After
‧
In-depth interviews are useful when you want detailed information about a person’s thoughts and behaviors or want to explore new issues in depth. Interviews are often used to provide
Table 6.2 is the type of interview which we used in this dissertation; focus group interview and in-depth interview; the interview details, experimental subjects and the definition are recorded. The following show the design principles and experiment details for every proposition. In addition, we also show the details interview questions one proposition by another.
(a) Design principles and experiment details for Proposition 1
The Proposition 1 is “the SMEs Classification Module of advertising mini story generator system which was developed based on the Maslow's Hierarchy of Needs could be used to construct the user-model structure as the recommender systems”. The objective of Proposition 1 is to make sure the classification of recommender systems is efficacious and could achieve the advertisement’s goal. To make the experiment result to be easy to understand and convincing, we intend to utilize the desire of follow-up reading to confirm if the classification is correct or not. If it is reasonable and feasible to establish the classification of recommender systems by the user’s behavior and information, users will get attracted by the story content. On the other
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
104
hand, the main idea of this experimental design is to understand the experimental subjects’ intention of reading the advertisement which triggered our system to verify if the classification of the recommender systems is efficacious. If the classification is efficacious, the experimental subjects’ intention of reading will stronger then the inefficacious one.
Figure 6.2 ARF Model Expanded for Interactive (to 12 levels)
From this perspective, the most basic evaluation is examining the effectiveness of the advertising story and focus on the internet advertising. Harvey (1997) proposed the ARF model expanded for interactive (Figure 6.2) including twelve levels in which three levels (Clickthrough, Interaction, and Attitude Shift) of the twelve levels are regarded as the persuasion stage. Harvey considered that the “Clickthrough” is an effective indicator in the persuasion stage.
On the other hand, Novak and Hoffman (1997) deliberated that standardizing the Web measurement process is a critical first step on the path toward the successful commercial development of the Web and advanced two indicators to measure:
exposure and interactive. The interactive indicator which is recording the interactive behavior between user and the web pages measures the effectiveness of advertising.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
105
Based on the mentioned theories, we divided the story into five sections and used 30 people to do the basic test (Figure 6.3). After the user finish filling their basic information, the web page will show the first paragraph of the story. User can choose if they want to read the next page or not by the “next” or “leave” button and the interactions will be recorded. The starting value is 0. If user moves on the next section, the value will be plus 1. (The highest score is 5 and the lowest score is 1).
Figure 6.3 The text process with five sections
Figure 6.4 Questionnaire input
Figure 6.4 is the system screen which displays the questionnaire of the system.
After pressing the enter button, the first paragraph of the story will be displayed with two selection buttons that let user chooses whether they want to go next page or leave
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
106
the system (Figure 6.5).
Figure 6.5 The web page shown as one of story paragraph
It's worth noting that the 30 experimental subjects we chose are the MBA graduate students and may become SME’s owners in the future to test the system preliminary. To make sure the integrity and correctness of this verification, we also coordinate the interviewing directly with Ilan local SMEs which we mentioned before to support the verify results.
In addition, the three click rule proposed by Zeldman (2001) means that a user of a website should be able to find any information with no more than three mouse clicks (otherwise, they will leave the site). On the other hand, Moriarty (1983) considered the effects of advertising are manifested by education, perception and persuasion.
Hence, we select 3 as an effect threshold, which means if user interacts with the site more than three click times, they think the content is useful for them which could be called the effect of advertising. Concluding, if a user’s click times (the score) is more then 3, we can then say it’s an efficacious advertising. Based on this result, we can deduce that if user be attracted by the story content, which indirectly means the
‧
classification is correct and utility the user behavior and information to establish the user-model which is based on the Maslow's Hierarchy of Needs is legitimate.
Table 6.3 The interview questions for Proposition 1
Focus Group Interview In-Depth Interview
Habits purpose of these questions wants to ask the habits and customs about the experimental subjects to support the computer using behavior which is used to test the effectiveness of internet advertising. The aim of the above question is to understand the relationship between the using behavior and interest to ensure the premise of our test is credible (If user has inclination to click the web page, then we can say they think the content is useful for them and could say it is effect advertising). The conclusion will be given in
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
108
the Section 6.3.
The following are another 2 tables (Table 6.4 and Table 6.5) describing the interview questions for the other propositions and have the same format as Table 6.3.
Each of table has two dimensions of the question type (habits and customs of the experimental subjects and system related), two dimensions of interview type (Focus Group Interview and In-Depth Interview) and the interview time shown in the table below. In Section 6.3, we will use these three tables to encode the interview data, and that’s the reason why we explain the table format first.
(b) Design principles and experiment details for Proposition 2
The Proposition 2 is “the recommendation from the advertising mini story generator system could encourage SMEs to follow the procedure of ImageCons, based on the user-model construction which construct by the Maslow's Hierarchy of Needs”.
The aim of the Proposition 2 is to test and verify the effectiveness of the advertising by observing the user’s aspiration of using the ImageCons which publicize by the advertising content. It means that we want to connect the experimental subjects’
opportunity of innovation and change to the advertisement provided by our system.
By imitating their past experience which prompts the experimental subject to do conversion in their business or daily life to make a new advertisement to our system to stimulate them to do service innovation (another new conversion). From another perspective, this is a validation method which through observing the past experience of the experimental subject to speculate that if they will be motivated by the advertisement of our system or not.
On the other hand, in this part we will demonstrate some test advertisement and inquire about their willingness of using ImageCons or not. Through the habits and customs of the experimental subjects and the actual test of the system, there is some
‧
interview question related to the Proposition 2 shown in Table 6.4.
Table 6.4 The interview question for Proposition 2
Focus Group Interview In-Depth Interview
Habits and Customs
None
1. What’s your opportunity to do the transition?
2. You said that the government has conducted some class to guide you to do transformation. Please describe the class which you told us to give advice of the the other model example and describe it?
5. What’s the reason to make you to use the internet to publicize your business?
System
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
110
There are two types of questions in the two interviews; habits and customs of the experimental subjects and system related; in Table 6.4. The aim of the system related question of Proposition 2 wants to understand the experimental subjects’ willingness to use ImageCons, and the habits and customs question want to connect the daily habits of experimental subjects with the story plot to speculate if the advertising is efficacious or not (for the reason that we ask the question which focuses on the transformation motivation process of the experimental subjects, like: stimulated by government class or the other trade). The completion will be discussed in the Section 6.3.
(c) Design principles and experiment details for Proposition 3
On the other hand, the Proposition 3 is “the SMEs will have more incentives when there are more their own related elements in the story”. As same as Proposition 2, Proposition 3 could be proved through observing the reaction of the experimental subject: SMEs. The authentication methods of Proposition 2 focus on the interview of their attitude of daily life, but the Proposition 3’s interview process unfolds by experimental subjects use the system by themselves and observing the reaction of experimental subjects during the using period. There are two selections in the system process in Proposition 3: with the personal related element and without the personal related element. Based on the determination of experimental subjects making choices about which ones do they want to read, the inquiries about how they feel about this test will then be conducted.
Figure 6.6 shown after the questionnaire page (Figure 6.4) is the system screen demonstrating two choices that let user make a selection; with the personal related element or without the personal related element. After user determines which one does they prefer, system will show the page based on their choice and record to
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
111
support Proposition 2
Figure 6.6 An exemplar of the two choices
Table 6.5 shows the interview questions for Proposition 3, which are separated in two parts as did for Proposition 2; habits and customs of the experimental subjects and system related. In addition to ask the experimental subjects directly which one they like, personal interrelated element story or not? We also find some daily habits to support our Proposition 3 as their reading habits and purchasing habits. The results will be shown in the Section 6.3.
Table 6.5 The interview question for Proposition 3
Focus Group Interview In-Depth Interview
Habits and Customs
None
1. What kind of industry do you want to observe?
2. Do you particularly pay attention to their industry-related news?
3. Please describe your attitude of an uncertain situation.
‧
4. Please describe a selling process situation that you do not accept.
The above questions (Table 6.3, Table 6.4 and Table 6.5) are the main interview questions leading principles and were adjusted by the circumstances at the time.
Through the several case interview to help to understand the other similar object and industry in order to popularize our system to the other SMEs in the different Recreational Agriculture Area. To strengthen the validity of the examination, we establish the case study database which content interview recording and transcripts to support the examination.