CHAPTER 6. EVALUATION
6.3 E XPERIMENT D ETAIL
6.3.2 Experiment Design
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
studying in graduate school or graduated in the past three years and now working in startups or design departments. All of the subjects are aged from 22 to 28 and have basic knowledge and practical experiences in service design.
6.3.2 Experiment Design
In order to get the data for justifying our propositions, we design an experiment containing four phases from the experimental subjects’ point of view and we conducted the experiment and collected data from 2013.06.10 to 2013.06.24.
Figure 6.1 Experiment Process from the Subjects’ Point of View
Phase1: Framing by own experiences
At first, the subjects were given a case as the brief of the design challenge (Appendix B) and were asked to do concept mapping to frame the situation based on their own experiences (Appendix D and E). The aim of this phase is to derive one or more insights.
Phase2: Framing with inspirations provided by Discover+ and Google
In this phase, the subjects can use Discover+ and Google Search to seek inspirations to reframe the situations. They can freely switch between two systems and add frame parts or new connections to their concept map as they wish. In order to recognize which parts and connections are newly added in this phase, they are request to use another color to revise the concept maps. At the end of this phase, they will come up with new insights whether are the revised versions of previous ones or totally new ones. Besides, they are asked to record two
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
kinds of data. The first is the portion of the usage of Discover+ and Google Search in this phase. They will give a percentage of it, for example, 70% for Discover+ and 30% for Google Search. Another data to be recorded is the portion of the source of the inspirations in these two phases. They will give a percentage as well. For instance, 40% from Discover+, 20% from Google Search, and 20% from own experience. As a result, we can do further analysis of the capability of facilitation of Discover+.
Phase3: Labeling the types of associations
After two rounds of concept mapping, there will be a concept map with lots of nodes and lines representing the relations between concepts. In phase three, the main purpose is to find out the portion of every kinds of association designers use when doing design synthesis. First, we will give the definitions and examples of the five types of associations that we categorized in Table 4.2. And then the subjects are asked to label the type of associations of all the connections on their concept maps if they think the relationship belongs to one or more type of associations we concluded. The reason why we ask the subjects to do the labeling task instead of doing by ourselves is because that the person who build up the map should be the one who understand the map and their cognition best. Rather than guessing, it’s better to let designers label them by themselves.
Phase4: Filling questionnaire
In the final phase, the subjects are requested to fill questionnaires (Appendix B). There are three purposes of the questionnaire. The first one is to identify the profile of the subjects in order to analyze the impact of different professions. Second, we want to collect how the subjects think about their method to do design synthesis. For instance, we ask the subjects
‘Do you think it is important to engage yourself in the environment when framing design context?’ At last, we ask some questions to examine the ability of facilitation of the system as
‧
five types. To verify whether these five types include all of the association ability that service designers use to do design synthesis, we asked the subjects to label the connections between the concepts on their concept maps (Appendix F). According to the collected data, all of the associations can be categorized into at least one of these five types of designers’ competence of association. Since there is no edge on the concept maps that does not fall into these five types of associations, we argue that the abilities required to form and process mental imagery are these five types of association.For the composition of the usage of these abilities, we can find a statistic in Figure 6.2.
The most used association is context association. It accounts for about half of the association.
Besides, similarity and contiguity association are also used commonly. The percentages are 16.3% and 21.6%. At last, analogy and contrast association are seldom used. There are only 5.7% of relations are analogy association and another 2.5% are contrast association.
Figure 6.2 Composition of each type of associations Context
‧
association competence between designers and non-designers, we analyzed the portion of the subjects using each type of associations. The percentages represent how many of the subjects use that type of association. For example, 6 out of 15 subjects of designers group used contiguity association, so the percentage is 40%. We can find that there is no significant difference in context and similarity associations between the two groups of subjects.However, non-designers like to list series of concepts to make their views more holistic so that their percentage of using contiguity association is higher than designers. For example, when concerning transportations in travel, they will try to list all the possible transportation tools like airplanes, trains, buses, cars, scooters, bikes and etc. On the other hand, more designers use analogy and contrast associations trying to change perspectives to view the design context. For instance, when thinking of the luxury ways to travel, they may also think of budget travelers. Also, when thinking of different types of tourism, they may make an analogy of the movie – ‘Up in the air’ to facilitate their thinking.
Figure 6.3 Percentages of using associations for different group of subjects
Look into the composition of associations on the concept maps (Figure 6.3), we can also
6.7%
‧
non-designers while non-designers use more contiguity associations than designers.r
Figure 6.4 Composition of each type of associations for different group of subjects
We infer that because designers have been trained more to have empathy, they tend to have objective mindsets trying to view things perceptually in many different perspectives. In contrast, for non-designers in this experiment including people whose major are business and engineering, they are more likely to have more subjective mindsets and view things rationally.
Hence, they like to keep their existing way of thinking and try to make their thought more logical and comprehensive. With perceptual mindsets, designers use more analogy to make their imaginations more concrete as the earlier example of using a movie to analogize a type of travel. Also, designers think more about the emotion of people, so they use more contrast associations to think of different experiences; for example, the happiest experience and the worst one in travel. On the other hand, due to the expertise in business or engineering, non-designers tend to have realistic thinking. They use more contiguity associations to list all the existing related factors to examine the feasibility.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Result for proposition 2
To justify proposition 2, we investigate the insight qualities in both macro view and micro view as we mentioned in Chapter 2.
For proposition 2-A, we examine the insight qualities in macro view with the insight depth map. First, we encode the insights derived from the experiments and fit them into the insight depth map. For example, an insight – ‘Travel agencies need to provide flexible travel package because it can attract more backpackers’ will be coded into (Functional, Service Provider) since it describes the economic value from provider’s perspectives. The same, another insight – ‘Young people need to travel often, because travel can make people grow up’
will be coded into (Intrinsic, Customer). The mapping result of the first insights derived from phase one of the experiments is shown in Figure 6.5. The numbers represent the subjects’
sequence number in the experiment. We can find that most people start from customers’
perspective while there are still some from the providers’ perspective.
Figure 6.5 The insight depth map of the insights in phase 1
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
And then we also encode the insights derived from phase 2 of the experiments and fit them into another insight depth map as shown in Figure 6.6. The bolded numbers with stars represent the subjects whose insights in phase 2 are deeper than in phase 1. Namely, these insights have a larger influencing scope in terms of the value or stakeholders’ perspective.
We can find that 56.7% of subjects (17 out of 30) enhanced their insight depth after reframing in phase 2 while there is no subject whose insight has a smaller influencing scope.
Figure 6.6 The insight depth map of the insights in phase 2
There seems no significant difference of the numbers of subjects whose insights go deeper between the subject group of designers (9 subjects) and non-designers (8 subjects).
However, analyzing their insights deeper, we can find that most of deepened insights of the non-designer group are from functional values to intrinsic values (5 out of 8) while designer group are more diversified, for instance, from economic values to functional values or from
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
providers’ perspective to customers’ perspective. Moreover, we also found that most of non-designers derived enhanced insights in phase 2 based on some context as in phase 1 (for example, same user perspectives or similar environments). On the other hand, most of designers found totally new insights in phase 2. To look into the causes, we can find some clues in the way the two groups do concept mapping. Designers tend to develop their concept maps evenly in many aspects. They wouldn’t start to derive insights until their concept maps are almost completed. It is like the strategy of breadth-first in graph search. Oppositely, non-designers construct concept maps in the manner of depth-first. They tend to have some intuitive assumptions or the direction they want to develop the service right after they read the design brief. They will firstly go into the direction of their assumptions when doing the concept mapping. Therefore most of their derived insights are surrounding their interested topics. So do most of their reframed insights in phase 2 are enhanced ones based on phase 1.
Appendix D and E give examples of concept maps by designer subjects and non-designer subjects. We can find that high portion of designers’ concept maps are evenly spread while many of non-designers’ maps are skewed (Appendix D and E).
For Proposition 2-B: To examine the insight quality from a macro view, we asked the subjects to score the qualities of their insights based on their perceptions. They scored the insight qualities in three dimensions including the degree of innovativeness, the degree of integrity, and the degree of agreement as we discussed in Chapter 4 (Table 4.2). For each dimensions, the subjects choose their perceptions of the derived insights in phase 2 comparing with the ones derived in phase 1 of the experiment. The options are ‘Better’, ‘No difference’ and ‘Worse’ that represent for score of 3, 2, and 1. For example, if a subject felt his insight in phase 2 is more innovative than in phase 1, he will choose ‘Better’ for innovativeness and get 3 points of score. Under this evaluation, we have a hypothesis that is
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
designed artifacts) insights is better than the qualities of the derived insights based on subjects’ own experience’. The data of subjects’ scoring is in Appendix C.
We use statistic software – PASW (Predictive Analytics Software), which was named SPSS (Statistical Product and Service Solutions) before and acquired by IBM in 2009, to facilitate the calculation. Table 6.1 shows the result of the descriptive statistics of the data.
We can find that the mean scores of the three dimensions are 2.50, 2.80 and 2.73 with the standard error are 0.093, 0.074 and 0.082 and standard deviation are 0.509, 0.407 and 0.450.
Furthermore, the minimum score of these three dimensions are all equal to 2 (Table 6.1). In other words, all of the subjects gave at least 2 points of scores. As a result, we almost can justify that our hypothesis is established.
Table 6.1 Descriptive Statistics for Proposition 2
N Minimum Maximum Mean
Statistic Statistic Statistic Statistic Std. Error
Innovativeness 30 2 3 2.50 .093
Integrity 30 2 3 2.80 .074
Agreement 30 2 3 2.73 .082
Valid N (listwise) 30
However, to be more rigorous, we further use another statistic method to test the hypothesis. Since we have 30 simple random samples with unknown standard deviation, we decide to use One-Samples T test as the method to verify the hypothesis. Hence, we have a null hypothesis and an alternative hypothesis as shown in Figure 6.7.
‧
Figure 6.7 Hypothesis of the testing
Table 6.2 shows the formula and interpretation of one-sample T test. There are two methods that we can use to test the hypothesis.
Table 6.2 Formula and interpretation of one-sample T test
𝐻!: 𝜇 < 𝜇!
First, we use PASW to do the One-Sample T test. We use 95% confidence interval of difference to do the test and the consequence is on Table 6.3.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Table 6.3 The result of One-Sample T test for proposition 2
Test Value = 2
t df Significance (1-tail) Mean Difference
95% Confidence Interval
of Difference
Lower Upper
Innovativeness 5.385 29 0.000004349732 .500 .31 .69
Integrity 10.770 29 0.000000000006 .800 .65 .95
Agreement 8.930 29 0.000000000403 .733 .57 .90
Using critical value method, we have to find out the t value and the critical value (c).
Figure 6.8 is a table of critical value of t distribution. We can find the critical value 𝑐 = 𝑡!,!!! = 𝑡!.!",!" = 1.699. Besides, in Table 6.3 we know the t value of the three dimensions are 5.385, 10.770 and 8.930. All of the t value are greater than c (5.385 > 1.699, 10.770 > 1.699, 8.930 > 1.699). Hence, the null hypothesis 𝐻! is rejected, which means 𝐻! is established.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 6.8 Critical Value of T distribution
On the other hand, we also test the hypothesis using confidence interval method.
According to the hypothesis, we know that 𝜇! is 2 and we are going to find out a. For the innovativeness, according to the formula (Table 6.2), we can calculate and derive that the value of 𝑡!,!!! * s/√n is 0.19 by calculating 0.5 (mean difference) - 0.31 (Lower Bound under 95% confidence interval). Following we use sample mean (x bar = 2.5 in Table 6.1) to minus 𝑡!,!!! * s/√n (0.19 as above calculation) and we can derive the value of a which equal to 2.31. And then we can reject 𝐻! and 𝐻! is established because 𝜇!(2) < a (2.31).
That is, based on our proposition, the insights quality in phase 2 is better than that in phase one in the innovativeness dimension.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Also, we do the same calculation for the integrity and agreement to see whether the insight qualities are also better for these dimensions. For the integrity dimension, the value of 𝑡!,!!! * s/√n is 0.15 by calculating 0.8 (mean difference) - 0.65 (Lower Bound under 95%
confidence interval). The value of a is 2.65 by calculating 2.8 (x bar, see Table 6.1) – 0.15(𝑡!,!!! * s/√n). Hence, we can reject 𝐻! and establish 𝐻! since that 𝜇!(2) is smaller than a(2.65). We can say that the insight qualities are also better in phase 2 for the integrity dimension.
At last, for the agreement dimension, the value of 𝑡!,!!! * s/√n is 0.163 by calculating 0.733 (mean difference) - 0.57 (Lower Bound under 95% confidence interval).
The value of a is 2.567 by calculating 2.73 (x bar, see Table 6.1) – 0.163(𝑡!,!!! * s/√n). As a result, the 𝐻! is also established since 𝜇!(2) is smaller than a(2.567).
In order to ensure the level of significance, we examine the p-value (the Significance (1-tail) in Table 6.3) which are 0.000004349732, 0.000000000006, 0.000000000403 and find that they are all much smaller than the α(0.05). As we know, if α > p-value then it reach the level of significance. Hence, we can conclude that the result of the above three testing is very significant.
To sum up, the testing shows that the qualities of insights derived in phase 2 are better than that derived in phase 1 in all of the three dimensions. However, follow the instruction of the experiment, the subjects can use both Google Search and the designed artifact – Discover+ to seek for inspiration. In order to examine the facilitation of the artifact, we also collect the using statistic from the subjects. The subjects write down the percentage of the source of inspirations and the results are shown in the Figure 6.9.
‧
Figure 6.9 The portion of source of inspirations (insight qualities in macro view) (phase 2).
According to the figure, we can find that averagely 85% of the inspirations come from Discover+ and the other 15% come from Google Search. Furthermore, we also examine the average of the subjects whose insights go deeper and not go deeper in phase 2. Discover+
inspired the subjects who derived deeper insights for 92% while Google Search only for 8%.
On the other hand, for the subjects whose insights didn’t go deeper in phase 2, they were inspired by Discover+ for 75% and by Google Search for 25%. We found that the subjects who deepen their insight depths in phase 2 perceived more inspirations from Discover+ than others whose insights are not deeper in phase 2.
Besides, in the micro view, we also examine the difference between the subjects who have higher perceived satisfaction and the ones with lower. Since all the subjects gave at least two points to all of the three dimensions, which means they think the new insights in phase 2 were not worse than in phase 1, we set the standard of 7 points of the total perceived
‧
satisfaction score in all three dimensions. In other words, subjects with total score above 7 points, which means that the subjects have better satisfactions in at least two dimension, are seen as the group that have a higher perceived satisfaction. On the other hand, the subjects with the score below 7 are seen as the group that did not perceive a better satisfaction of insight qualities. The results (Figure 6.10) show that the subjects with higher perceived satisfactions were inspired by Discover+ for 91% and by Google Search for 9%. As for the subjects with lower satisfactions, they were inspired by Discover+ for 66% and by Google Search for 34%. It can then be inferred that the more the subjects were inspired by Discover+, the better the insight qualities will be.
Figure 6.10 The portion of source of inspirations (insight qualities in micro view) (phase 2).
Furthermore, Figure 6.11 and Figure 6.12 show the total portion of source of inspirations for the whole concept mapping process (both phase 1 and phase 2 of the experiment). We also can find that the subjects who have better insight qualities (no matter in macro view or in micro view) were inspired more by Discover+ than Google search. At the mean time, the
Source of insirations Perceived
satisfaction
‧
other subjects whose insight qualities do not be better were inspired less by the artifact but more by Google search or their own experiences.
Figure 6.11 The portion of source of inspirations (insight qualities in macro view)(phase 1 and 2).
Figure 6.12 The portion of source of inspirations (insight qualities in micro view)(phase 1 and 2).
Discover+ Google Self-‐Experience
Source of inspiration Insights
deeper
Discover+ Google Self-‐Experience
Source of inspirations