EVALUATION OF VIRTUAL REALITY PRESENTATION IN USER TESTING PROCEDURE FOR PRODUCT USABILITY OF A CONCEPTUAL DESIGN

EVALUATION OF VIRTUAL REALITY PRESENTATION IN USER TESTING PROCEDURE FOR PRODUCT USABILITY OF

A CONCEPTUAL DESIGN

Chiuhsiang Joe Lin

* Wei-Jung Shiang

Rou-Wen Wang

Tse-Jung Chen

1

Department of Industrial Management National Taiwan University of Science and Technology

Taipei, Taiwan 106, R.O.C.

2

Department of Industrial and Systems Engineering Chung Yuan Christian University Chung Li, Taiwan 32023, R.O.C.

Key Words: conceptual design, computerized visual presentation, image- based virtual reality, usability study.

ABSTRACT

This study examines three different modes of visual presentation, pic- ture, animation, and image-based virtual reality (image-based VR), in order to find the most suitable presentation method for conceptual design with an emphasis on the evaluation of usability and user-centered design character- istics. In a user testing procedure, questionnaires were used to evaluate static picture, animated, and image-based VR presentation conditions and also the combination of these techniques for a task, which consisted of the per- formance of individual presentations and a combination of three presenta- tions. The results indicate that image-based VR is the best single medium for conveying the critical ideas of the conceptual design to product evalua- tors in a limited amount of viewing time. However, the use of pictures and animation in combination with virtual reality (VR) can display the concept- designed ideas more completely.

I. INTRODUCTION

Technological innovation plays a major role in creat- ing and maintaining competitiveness in the global market.

Enterprises have to develop new products continuously in order to ensure their competitive advantage. Human- centered design has a critical influence on whether new products are successful (Green and Jordan [1]). Customers tend to expect products that can meet both their functional requirements and usability needs beyond their satisfaction with basic requirements.

The early identification of an optimal concept is critical to the design process in order to increase the chances of satis- fying customers (Gironimo et al. [2]). The product design cost accounts for only about 5% of the total cost, but 80 to 90% of the cost is decided in the design stage of the entire

product life cycle (Boothroyd [3]; Gatenby and Foo [4]).

Designers usually repeatedly analyze design problems, con- ceptual design, embodiments of schemes, and a detailing stage to try out different solutions and finally come up with a new design. Particular emphasis is placed on the analysis of design problems and the conceptual design phase (Oh and Stuerzlinger [5]). The conceptual design phase in the initial product development stage is usually the key for de- fining the cost and the quality of the product. Problems identified early can be fixed at much lower costs and in much less time than those found in later stages, when the parts, dimensions, and cost have been determined (Ullman [6]).

Concept generation and selection are often dependent upon the experience and knowledge of designers, who produce as many different concepts as possible to generate a wide variety of ideas to evaluate at the next level of design.

* Corresponding author: Chiuhsiang Joe Lin, e-mail: [email protected]

On the other hand, the ideas adopted from the designer may not be able to meet customer needs and expectations, so it is hoped that designers will be able to avoid design deficiencies if users perform a product evaluation and pro- vide suggestions and feedback on the design before its manufacture. Conceptual design needs a tool that is easy to use, as most participants in product evaluations are not design software experts. This tool must be applicable to concept communication, information sharing, design form presentation, and so on because most ideas will undergo multiple iterations and reconditions.

To meet user needs, convenient functions are added to a product, but such additions usually increase the com- plexity of product use. The main goal of a usability eva- luation is to incorporate a series of tests into the design and development process of a product, the results of which can be integrated into the product development, and sufficient resources must be allocated to carry out the activities speci- fied. A usability evaluation is a communication bridge between users and designers in the product design and de- velopment stages. Although improvements to a user in- terface in terms of objective performance can improve a product, not all users will necessarily be thoroughly satis- fied with the product. Kansei engineering emphasizes the subjective assessment of usability and the translation of users’ impressions of and feelings about a product into de- sign elements (Nagamachi [7]). Therefore, usability eva- luation consists of objective performance and subjective impressions, which are considered equally important in design.

This study adopts viewing time and correctness of answers to questions as the objective performance indicators and adopts nine characteristics-accessibility, redundancy, stability, adjustability, flexibility, expansibility, maintain- ability, fitness, and degree of human-centered design-as subjective assessment categories in order to examine the usability of conceptual design. The degree to which users experience a product is important for usability evaluation.

If the product design is presented in such a way as to en- hance the interaction between user and product, users can understand thoroughly the design and operation of a product.

How a product is presented affects the usability evaluation, so the method of presentation is a noteworthy issue. A sketch of the design concept is presented initially in visual models created by designers. The sketch presentations are usually divided into static drawings or pictures, animation, or image-based VR.

In recent years, researchers have engaged in lengthy

debate on the opportunities for using animation in learning and instruction. In compare with conventional instruc- tional material that consisted of a diagram of an electrical circuit and was accompanied function illustration by text, Kalyuga et al. [8] indicated that static drawings or pictures only can increase the user’s cognitive load and cause ad- verse outcomes because of over-reliance on only one me- dium. The information might be not be communicated successfully to the user by drawings and pictures when the functions of a product are highly complicated. Animation, on the other hand, can augment the static pictures, in- creases the attention and motive of viewers, and has more explaining power. Previous studies have examined the differences and performance values of static drawings or pictures versus animation, and the results show a tendency that viewers with a visual cognitive style learn significantly better from animation than from static pictures (Blomé et al. [9]; Hidrio and Jamet [10]; Höffler and Leutner [11];

Höffler and Schwartz [12]). Höffler and Leutner [11]

conducted a meta-analysis of the effects of dynamic and static visualizations on learning outcomes through integrat- ing the findings of a large number of studies, calculating the overall effects, and identifying possible moderator va- riables. Höffler and Schwartz [12] developed four different versions of a computer-based learning environment on the role of surfactants that to dissolve dirt from a surface dur- ing the washing process to examine the effects of self- pacing versus system pacing in different versions of a computer- based learning environment (static pictures/animation).

Hidrio and Jamet [10] used the four-stroke engine as an example to assess the effectiveness of static and animated illustrations. Blomé et al. [9] followed ergonomic guidelines to evaluate an interactive multimedia system (hyperlinked text, pictures, and animation) and a conventional system (text and pictures). These studies compared the learning per- formance between static pictures and dynamic animation, but have not extended the comparison to the case of virtual reality, which has become an important tool in training and learning today.

Over the past decades, a considerable number of studies have been conducted on the development of VR technology in design and training applications (Crumpton and Harden [13]; Kuo and Wang [14]; Dong et al. [15]; Lin [16];

Bruno and Muzzupappa [17]; Kim et al. [18]; Noon et al.

[19]; Melemez et al. [20]). Li et al. [21] examined the

content of virtual experiences in e-commerce as concur-

rently verbalized by a sample of 30 participants while in-

teracting with four 3-D products. Thirteen different types

of psychological activities were observed and classified into five categories of virtual experience: active process, pres- ence, involvement, enjoyment, and affordances. It was found that virtual experience is vivid, involving active, affective psychological states in an individual interacting with 3D computer simulations. Besides, Lin [16] used a nail puncher as an example and, developed a collaborative conceptual de- sign integration system by using design for manufacturing and assembly (DFMA) for innovative conceptual design and filtering, and using VR technology for visualized col- laborative conceptual design communication. The results showed that this system reduced development costs and the number of parts, and it further shortened the product development cycle. Crumpton and Harden [13] also created a virtual representation of a cereal packing operation in order to explore the possibility of using VR as an instructional aid in ergonomics courses. The results showed that the visual representation of the work environment was ade- quate for performing ergonomic evaluations, demonstrating the usefulness of employing VR in ergonomic class instruc- tion. The above literature showed that VR technology had promised an effective and efficient method of conveying product conceptual design ideas to the user. For these reasons, three-dimensional (3D) product visualization with VR technology, a new form of rich media advertising that enables consumers to interact with a virtual product much as they would with a physical product, has received con- siderable attention.

However, few attempts have been made to compare the presentation of conceptual designs by VR technology with traditional presentations such as static pictures and animation nowadays. Alcaide-Marzal et al. [22] compared 3D virtual object and 2D sketches, which were representa- tions in perspective and presented in several views by ZBrush software. In this study, 2D sketches were symbolic representations of an object presented on the A4 paper with pencil drawing and 3D representation was a virtual reality object without any word annotation. In terms of quantity of information generated, designers need several 2D sketches to interpret their solutions, workflow and idea, but it takes just one 3D sketch to describe an object completely.

Tiainen et al. [23] explored benefits of revealing design ideas between virtual and physical prototypes of furniture products. The results found that the participants came up with more developmental idea when evaluating virtual pro- totypes than physical prototypes. Bruno and Muzzupappa [17] developed a system named VP4PaD (virtual proto- typing for participatory design) and used it in a microwave

oven design to examine the differences between the ‘‘user- real product’’ interaction and the ‘‘user-virtual product’’

interaction. The results indicated that the virtual interface neither increased the difficulty of understanding the product interface nor distorted the effectiveness of the interaction.

The above studies did not draw comparisons with other currently available media that might be more cost effective and more easily applicable to user testing procedures, such as showing pictures and animated clips to the participants in the usability evaluation. This study compared three dif- ferent media, static pictures, animation, and image-based VR, in an empirical user testing procedure to find appro- priate methods of presenting product conceptual design and effectively convey usability design ideas to the testing users. Utilizing a prototype design of an electronic ship console and aiming to enhance the usability of the console, the present study also evaluated whether the combination of the three media can be more effective than the single medium alone. Specifically, the prototype was designed based on the user-centered design approach, which consid- ers accessibility, redundancy, stability, adjustability, flexi- bility, expansibility, maintainability, and fit to the design.

The user testing procedure was performed to obtain user feedback on several usability characteristics of the ship elec- tronic console under design, including (1) the anthropom- etric fit to the target user population, (2) major components of the new electronic console, (3) details of operational devices, (4) ease of entering and exiting the seat of the new console, (5) the adjustability of the seat height and the front-rear adjustment of the panel, (6) the adjustability of the armrests, headrests, keyboard angle, and display angle, and (7) the replacement of the devices during maintenance.

II. METHOD

1. Apparatus and Materials

A new electronic console was developed in a ship construction company. The console must accommodate several computer displays, a keyboard, and other controls.

It was designed for use in a ship with consideration of

human operators working at sea. For this study, the con-

ceptual design materials were provided by the company in

the forms of pictures, animation, and VR. The framework

of the new electronic console was developed according to

the nine design characteristics listed above by engineering

specialists in the company, with a focus on reducing work-

load and improving performance. The nine characteristics

were considered when designing the new electronic con- sole as follows:

i. Accessibility:

During the design stages of the new electronic con- sole, it was for operators to be able to move into position and operate quickly to prevent damage from reaction time delays in emergencies.

ii. Redundancy:

If equipment of the new electronic console is dam- aged, it is necessary to replace it immediately in order to ensure that operations can continue.

iii. Stability:

For operators, in addition to increased physiological and psychological burdens, the difficulty of operation could also be increased by operation in difficult conditions. For example, manipulating a mouse could be difficult under heavy seas or inclement weather conditions. Providing handles for operators to use in maintaining their position could reduce the operator burden.

iv. Adjustability:

The new electronic console should to be designed such that the angle, height, and location of the display, the keyboard, and the chair can be adjusted by different op- erators to increase comfort and thus improve performance.

v. Flexibility:

The new electronic console should be able to change modes for the performance of tasks such as target acquisi- tion, fire control, electronic warfare, and anti-submarine warfare. In other words, the screen, keyboard, or operating interface should be able to meet different task requirements in order to increase the available space.

vi. Expandability:

To meet different operational requirements, it must be necessary to add or remove displays and operational de- vices such that the new electronic console can be adapted to a variety of warships and task types.

vii. Maintainability:

The maintenance of the new electronic console includes modular design and separation design for convenience and timeliness. The modular design allows new modules to be replaced directly should damage occur. In addition, the

modular module should be maintained by maintenance professionals to reduce maintenance time and remove the responsibility for maintenance from the operator. The se- paration design involves separation of the host and the control interface to avoid congestion at the electronic console.

With this design, the maintenance worker can repair dam- age to hardware or problems with software separately.

viii. Fitness:

Every part of the new electronic console (desk, back of chair, armrest, screen, work space, and so on) was de- signed according to suggestions by engineering profes- sionals to reduce the physical and psychological loads and the occupational injuries of the operators. Therefore, the height and width of the desk, height and angle of the inter- face, and the range of horizontal and vertical work fields were matched to more than 90% of the operators’ physical characteristics to ensure comfort during operation.

ix. Human-centered concept:

Influences from vibrations were reduced to improve the comfort and performance of the operators.

The conceptual design of the new electronic console was presented on a desktop computer (with Intel Pentium III, 952MHz, NVIDIA GeForce FX5200 128RAM graph- ics card) with a 15-inch monitor (AG NEOVO F-15 LCD).

2. Tasks

During the experiment, the participants had to view the design of the new electronic console using one of the three media: picture presentation, animation presentation, and image-based VR presentation. The static pictures de- scribed only the seated and exit conditions, as shown in Fig. 1. The animation presentation included about three seconds of video animation (15 frames per second) and showed the eight images in Fig. 2. The image-based VR pre- sentation allowed the user to adjust the angle and distance of viewing of the design, and to move or recombine the equipment of a new electronic console, as shown in Fig. 3.

3. Experimental Procedures

The study was conducted in a laboratory setting and

began with the administration of a short survey designed to

collect background information on each participant. The

participants were informed of the purpose of this study and

given background information on the product usage in the

ship. They were shown how to operate the computer inter-

face that would later show the pictures, animation, and VR

(a) before variances

(b) after variances

Fig. 1 The operation of exiting the electronic console described in only two pictures: (a) seated condition; (b) exit condition

Fig. 2 The animation consisted of fifteen frames showing an operator entering and exiting the console

(a) Original scene (b) Zoom in to view the details of the left-hand armrest

(c) Zoom in to view the details of the right hand

(d) Viewing different angles

Fig. 3 In the image-based VR presentation, users were able to (a) adjust the angle of viewing and distance, (b and c) zoom in, and (d) view from different angles

presentation of the prototype. The formal experiment began when the participants confirmed that they had no further questions about the experimental procedure or the equipment. The participants were asked to perform as they would in user feedback testing of product design.

They then began viewing the prototype of the new elec-

tronic console. During the course of each viewing, they

were asked to verbally present their questions about the

product usage, design, and anything that might be unclear

or confusing. All points about the conceptual design were

recorded, including questions, suggestions, affirmative im-

pressions, and criticisms by the participants. For those

Table 1 Sample questions about the questionnaire of approval degree of the usability

Characteristics Question Accessibility Can the operator quickly access the console?

Redundancy Does the console have redundant design?

Stability Can it provide a stable and light-load operation interface in vibration environment?

Adjustability Can the user adjust the equipment settings for more comfortable op- eration?

Flexibility Can the new console design be applied to the manipulation of the different types of combat?

Expandability Can the new console design be changed for different ship require- ments?

Maintainability-modular Can the new console replace damaged modules instead of field main- tenance?

Maintainability-separation Can the design reduce maintenance time and improve efficiency?

Fitness Can the console design fit the users’ physical requirements?

Human-centered concept Are design considerations of the new console human-centered?

Subjective rating: Ten ratings of the design attributes were given a 5-point Likert-type scale: (1) strongly disagree, (2) disagree, (3) neither agree nor disagree, (4) agree, and (5) strongly agree.

questions, the experimenter did not provide any explana- tions and responses. When the participant felt he or she had no more questions and was ready for the questionnaire session, the viewing time was recorded and the question- naire session was immediately begun. At the end of each viewing session, the participants then completed the test and usability questionnaires.

4. Questionnaires

Two questionnaires were used in the study. One consisted of 40 true/false questions that described the de- sign characteristics of the product to test the participants’

understanding of the design concept. The other was a usability questionnaire consisting of questions on a five- point Likert-type scale to evaluate their degree of approval of the usability design of the prototype. The second ques- tionnaire (Table 1) is common practice in usability testing or user feedback studies in product design. Since the second questionnaire was just a normal part of the user testing procedure and was not related to the purposes of this study, namely, finding an appropriate combination of presentation media, the results of the second questionnaire are not presented in this paper. Rather, the focus of the study was to determine which presentation method would best convey to the user the conceptual design of the proto- type. Therefore, the dependent variables of the present study were the viewing time, question counts (number of

raised questions), completion time of the questionnaire, and correctness of the answers to the questions, all of which were related to the degree of understanding of the design or the efficiency of conveyance of the design in- formation. The questionnaires were written in Visual Basic 6.0. The program recorded the time taken to complete them and the total scores of the questionnaires.

5. Data Analysis

Twenty-five participants aged 22 to 28 years were re- cruited for the study and provided informed consent. The primary inclusion criterion was that participants were able to operate computers with proficiency. All participants were healthy and reported no visual problems or diseases (e.g., motion sickness, color blindness, or physical disabil- ity) that could be detrimental to physical performance.

The participants were divided into three groups. Group A (9 participants) viewed three presentations of the prototype design in the following order: picture presentation, anima- tion presentation, and image-based VR presentation. Group B (8 participants) viewed only two presentations: the ani- mation presentation first and the image-based VR presen- tation second. Group C (8 participants) viewed only the image-based VR presentation. The task arrangement for the participants in this experiment is presented in Table 2.

From the above arrangement, three comparisons can

be made. When A-1, B-2, and C-3 are compared, the

Table 2 The experimental group assignment

Group Picture presentation Animation presentation Image-based VR presentation

A A-1 A-2 A-3

B B-2 B-3

C C-3

Table 3 Analysis of variances for question counts, viewing time, correctness of the answer to the questions, and completion time of the questionnaire in terms of the different presentation methods

Analysis of Variance for the comparisons of A-1, B-2, and C-3

Dependent variables Independent variables Mean Fn,m P-Value Picture presentation 15.89

Animation presentation 11.75 Question counts

Image-based VR presentation 10.37

F(2, 22) = 1.496 p = 0.246

Picture presentation 918.78 Animation presentation 760.13 Viewing time

Image-based VR presentation 664.00

F(2, 22) = 2.162 p = 0.139

Picture presentation 534.67

Animation presentation 455.13

Completion time of question-

naire

Image-based VR presentation 390.13

F(2, 22) = 4.604 p = 0.021

Picture presentation 91.11 Animation presentation 91.87 Scores of test questionnaire

Image-based VR presentation 91.87

F(2, 22) = 0.144 p = 0.867

presentation effectiveness of each of the individual me- dium can be tested, since each medium is viewed for the first time and only once. When A-3, B-3, and C-3 are com- pared, the performance of presenting three media (A-3) is compared to the performance of presenting two media (B-3) and one medium (C-3). It must be noted that the viewing procedure followed the order of pictures, anima- tion, and VR in the A-3 condition. The viewing proce- dure followed the order of animation to VR in the B-3 condition, and the C-3 condition had only VR. This com- parison is intended to test the additive performance of the picture and animation presentations, since it is usually easy to add these two to the user test without a significant time cost to the user.

Question count, viewing time, completion time of the questionnaire (answer time), and scores on the test ques- tionnaire (correctness of the answers to the questions) were collected as dependent variables. Question count was defined as the number of questions asked by participants when they were confused about the conceptual design of the new electronic console during the viewing task. View- ing time was defined as the time spent by participants

viewing the prototype design until they felt they had no more questions about it. Participants had to complete the questionnaire after the viewing task. The time spent com- pleting the questionnaire was recorded as the completion time of the questionnaire. Scores on the test question- naire were recorded as the scores of the test questionnaire.

The data of question count, viewing time, completion time of the questionnaire (answer time), and scores on the test questionnaire (correctness of the answers to the ques- tions) obtained from laboratory experiments were analyzed by two-way analysis of variance (ANOVA) with the Tukey HSD as the post hoc multiple comparison test.

III. RESULTS

1. The Effects of a Single Medium

Results of ANOVA on the comparisons of A-1, B-2,

and C-3 revealed no significant differences in viewing

time and question counts between presentation methods, as

shown in Table 3. The viewing times were 918.78, 760.13,

and 664 sec for picture presentation, animation presentation,

Table 4 Analysis of variances in A-3, B-3, and C-3 for question counts, viewing time, correctness of the answer to the ques- tions, and completion time of the questionnaire in terms of learning effect

Tests of Between-Subject Effects of A-3, B-3, and C-3

Dependent variables Independent variables Mean Fn,m P-Value

A-3

Picture presentation Animation presentation Image-based VR presentation

22.67

B-3 Animation presentation

Image-based VR presentation ^15.88

AB

Question counts

C-3 Image-based VR presentation 10.37

F(2, 22) = 4.37 p = 0.025

A-3

Picture presentation Animation presentation Image-based VR presentation

1633.78

B-3 Animation presentation

Image-based VR presentation ^1275.13

A

Viewing time

C-3 Image-based VR presentation 664.00

F(2, 22) = 12.67 p = .000

A-3

Picture presentation Animation presentation Image-based VR presentation

90.55

B-3 Animation presentation

Image-based VR presentation 91.87 Scores of test

questionnaire

C-3 Image-based VR presentation 91.87

F(2, 22) = 0.18 p = 0.840

and image-based VR presentation, respectively. The ques- tion counts were 15.89, 11.75, and 10.37 points for picture presentation, animation presentation, and image- based VR presentation, respectively.

In addition, no significant differences between pres- entation methods were found in the scores on the true/false questions (F (2, 22) = 0.14, p > 0.05), as shown in Table 3.

The scores on the true/false questions were 91.11, 91.87, and 91.87 for picture presentation, animation presentation, and image-based VR presentation, respectively. In con- trast, the results indicated significant differences between presentation models (F (2, 22) = 4.6, p < 0.05) in comple- tion time of the questionnaires, as shown in Table 3. The completion time of the questionnaire was longer for picture presentation (534.67 sec) than for animation presentation (455.13 sec) and image-based VR presentation (390.13 sec).

2. The Effects of Combined Viewing

The data of viewing time and question counts were collected in the viewing order of the picture presentation, animation presentation, and/or image-based VR presenta- tion in order to clarify the effects of combined viewing on understanding of the design conception. The total question

count (22.67 points) presented by participants in Group A was summed from the picture presentation (15.89 points), animation presentation (4.11 points), and image-based VR presentation (2.67 points). The total viewing time (1633.78 sec) spent by participants in Group A was summed from the times of picture presentation (918.78 sec), animation presentation (288.89 sec), and image-based VR presentation (426.11 sec). In addition, the total question count (15.88 points) presented by participants in Group B was summed from the animation presentation (11.75 points) and image- based VR presentation (4.13 points). The total viewing time (1275.13 sec) of participants in Group B was summed from the times of animation presentation (760.13 sec) and image-based VR presentation (515 sec). Finally, the total question count of the participants in Group C, who viewed the prototype design in image-based VR presentation, was 10.37 points, and the total viewing time presented was only 664 seconds.

The above results were used for the comparison of

A-3, B-3, and C-3. There were significant differences in

question counts (F (2, 22) = 4.37, p < 0.05) between pres-

entation methods, as shown in Table 4. The total question

counts of the participants in A-3 (22.67) and in B-3 (15.88)

were higher than that of the participants in C-3 (10.37).

In addition, there were also significant differences in view- ing time (F (2, 22) = 12.67, p < 0.05). The viewing times of participants in A-3 (1633.78 sec) and B-3 (1275.13 sec) were shorter than those of participants in C-3 (664 sec).

There were no significant differences between presentation methods in scores on the true/false questions (F (2, 22) = 0.18, p > 0.05), as shown in Table 4. Scores on the true/

false questions were 90.55, 91.87, and 91.87 for A-3, B-3, and C-3, respectively.

IV. DISCUSSION

1. The Effects of a Single Medium

In this study, the conceptual design was presented ei- ther to be appreciated by viewers or to collect a variety of opinions on the design for the improvement of the design considerations. This study further examined the relation- ship between image-based VR presentation, animation pre- sentation, and picture presentation. One of the aims of this study was to examine whether presentation of the concep- tual design by the image-based VR technology was supe- rior to presentation by picture and animation in terms of con- veying the design to the viewer. Although our ANOVA results showed no significant differences in the question counts and viewing times, the descriptive statistics in terms of the question count and viewing time for the image-based VR presentation (10.37 points; 664 sec) were lower than those of the picture (15.89 points; 918.78 sec) and anima- tion presentations (11.75 points; 760.13 sec), and the pic- ture presentation was the worst. Our results are consistent with those of previous studies that found animation to be su- perior to pictures (Hidrio and Jamet [10]; Höffler and Leut- ner [11]; Höffler and Schwartz [12]). Höffler and Leutner [11] compared the effects of dynamic and static visualiza- tions on learning outcomes, and revealed an overall advan- tage of instructional animation over static pictures. Höffler and Schwartz [12] examined the effects of self-pacing ver- sus system pacing in different versions of a computer- based learning environment (static pictures/animation).

Although the ANOVA results indicated that animations were not significantly superior to static pictures, the results of interaction effects had shown that learners had better results when using animation rather than static pictures in the self-pacing condition. In addition, Hidrio and Jamet [10]

provided four conditions including no pictorial information, a single static picture illustration, multiple illustrations,

and animation in order to assess the effectiveness of static and animated illustrations. The results showed that ani- mation did improve the participants’ mental models.

Similarly, Blomé et al. [9] found the understanding of the requested information was good with the interactive mul- timedia animation system compared to other more static system. From what has been discussed above, animation can improve over the problems of static pictures, such as the discontinuity of different drawings and its inability to show the actual operations.

In terms of VR presentation, our results showed that the VR presentation was the best, although the results were not statistically significant. Alcaide-Marzal et al. [22]

prepared experimental tasks involving producing as many solutions as possible to a given design problem by means of sketching. The participants were asked to design prod- ucts according to several photos of the style and functional requirements. After sketching, they were asked to talk about their design on number of solutions, workflow and idea generation. The results found the amount of infor- mation generated in 2D sketches were smaller than that in 3D sketches. The results of question count in our study is in agreement with Alcaide-Marzals’ finding [22], despite the statistical non-significance of our study that may have been due to the limited number of test participants.

A detailed review of the questions presented by par- ticipants who viewed the image-based VR presentation revealed that the depth and characteristics of those ques- tions were more informative than those of the others.

This may be attributed to the way the participants obtained the design information. The questions on the picture pre- sentation were mostly relevant to direct characteristics, such as the angle of the armrests, which could be adjusted horizontally and vertically. However, some of the ques- tions on the image-based VR presentation were more rele- vant to indirect characteristics that could not be viewed directly from the picture, such as feedback that the angles of adjustment of the armrests could be insufficiently large and the height should also be adjustable, as well as ques- tions about the material of the back of the chair. In other words, the questions about and comments on the VR pres- entation were found to be more informative than those of the other methods. In some cases, the participants even provided several viewpoints never considered by the de- signers. Our findings are in line with Tiainen et al. [23]

using furniture (sofa, shelves and tables) as sample to

compare virtual prototypes to physical prototypes. Their

results showed that virtual prototypes were difficult to

detect actual size of products but were more appealing than the physical prototype. In addition, the amount of developed ideas in virtual prototypes was higher than physical prototypes. In our study, the image-based VR pre- sentation can stimulate participants to generate more de- velopment ideas in the conceptual design stage. In addition, some of the participants in Group A (who viewed only the picture presentation) felt the picture presentation was not stimulating enough to make them want to view the design.

Our study suggests that the picture presentation contained the least information and was less stimulating than the other two types of presentation.

The results of ANOVA showed no significant differ- ences between the presentation methods on the scores of the test questionnaire, but the descriptive statistics on the scores of the test questionnaire for the VR presentation (91.88) and animation presentation (91.88) were the high- est, indicating better understanding of the design concept than with the picture presentations (91.11). The results showed significant differences in the completion time of the questionnaire, with participants who viewed the VR presentation (390.13 sec) spending significantly less time than those who viewed the picture (534.67 sec) and anima- tion presentations (455.13 sec). It was also found that the participants who viewed the VR presentation spent the least amount of time scanning the conceptual designs yet demonstrated the best performance. The most likely ex- planation for this contrast lies in the nature of the VR pre- sentation, which can provide plentiful information for the participants to imagine the considerations of the design more easily. In other words, it was difficult for participants who viewed the picture presentation fully to reconstruct the framework of the conceptual design from the pictures, as revealed by Kalyuga et al. [8] research, so they required more time to answer the questions. Clearly, the findings support that for participants who viewed the design for the first time, the VR presentation was more effective in as- sisting people to understand the design concepts. From what has been discussed above, it appears reasonable to suggest that presenting a conceptual design by the image- based VR is better than presenting it with the other two media, and that the image-based VR presentation would allow designers to collect more pragmatic advice from viewers within a limited viewing time.

2. The Effects of Combined Viewing

It is also important to investigate whether combining the three media was more effective than using a single

medium. For this purpose, this study further compared Group A, who viewed the picture presentation, animation presentation, and image-based VR presentation; Group B, who had viewed the animation presentation and the image- based VR presentation; and Group C, who viewed only the VR presentation.

The results of ANOVA showed significant differences in the total question counts and the total viewing times.

The total question count for participants in Group C (10.37 points) was significantly lower than that for the other two groups (Group A: 22.67 points; Group B: 15.88 points).

This was reasonable, since each medium added further information to that gleaned from the previous one and pro- mpted different questions. Therefore, the more the stages of viewing, the more questions were posed. Even after Group A had viewed the picture and animation, some more questions on the design in the final phase of viewing of the VR presentation (A-3) were still raised for clarification.

This appears to indicate that the VR presentation provided information that was not available in the picture and ani- mation presentations, prompting further questions from the participants in Group A and Group B. These results showed that the combination of presentation methods certainly had an impact on increasing overall understanding of the con- ceptual design. Based on this finding, it seems reasonable to recommend viewing all three presentation types if the user has sufficient time and cost is not an issue.

3. Viewing Efficiency

However, if the concern here is to be cost effective, is it necessary to view every medium in order to get the whole design idea? Or is it efficient and sufficient to view just one? If so, which one? Note that in comparing single view- ing and combined viewing, no significant differences in test scores were found. The participants in Group A spent the most time viewing all three presentations, yet their scores on the true/false questions were not significantly different from those of the other groups. In other words, the extra time spent on the pictures and animation did not further improve the participants’ level of understanding of the design concept beyond that attained with a single view- ing of the VR presentation. On the other hand, the par- ticipants in Group C, who viewed the image-based VR presentation, achieved the same score on understanding of the design concept. This suggests that a single viewing of the VR presentation is more cost-effective. Furthermore, the question count also revealed something interesting.

The question count for participants in Group A was reduced

gradually in the order of picture presentation (15.89 points), animation presentation (4.11 points), and image-based VR presentation (2.67 points). In Group B, the question count was also reduced gradually in the order of animation pres- entation (11.75 points) and image-based VR presentation (4.13 points). It can be seen that the users asked the most questions when a medium was first viewed, and that there- after, the number of questions fell significantly, indicating that any initial confusion was reduced after further stages of viewing through different media. Additional viewing is helpful in this regard. However, if only one medium is to be selected, VR is suggested, since the question count for the VR presentation was the lowest of the three media (15.89, 11.75, and 10.37 for picture, animation, and VR, respectively), indicating that VR prompted fewer questions on the design concept.

V. CONCLUSIONS

The concept-design is presented in hopes of obtaining feedback on possible deficiencies in a design. It can also be used to obtain affirmation from the observers. Therefore, when time and money are limited, it is important to select the presentation method that is most efficient and most effective. The designer hopes to glean useful suggestions and affirmation on the design from the viewer. The best information and depth of the comments and feedback in this study were found to be prompted by the image-based VR presentation method. We suggest adopting the image- based VR presentation method if the designer hopes for observers to provide in-depth suggestions in the shortest time possible in the conceptual design stage. In addition, this study has verified that combined viewing can have an impact on the understanding of concept-design. It can also achieve superior performance in conveying understanding of the conceptual design, although the participants may spend more time viewing the design to clarify their prob- lems and confusion when methods are combined. The concept-design was presented with a more advanced pres- entation method not only to encourage people to view the same design again but also to allow people to glean and sup- plement information that they previously had not noticed.

We suggest that future studies examine the effects of immersive VR technology on understanding of concept- designs and determine suitable modes of presentation of specific designs. In addition, this research addressed only the “visual” side. Many experiences, such as the design cha- racteristics of stability and maintainability in our example,

are difficult to precisely and sufficiently evaluate through only visual perception. Some forms of physical simula- tion would be more helpful if provided to the participants.

REFERENCES

1. Green, W. S., and P. W. Jordan, eds. 1999. Human Factors in Product Design: Current Practice and Future Trends. London: Taylor and Francis.

2. Di Gironimo, G., A. Lanzotti, and A. Vanacore. 2006.

“Concept Design for Quality in Virtual Environment.”

Computers & Graphics 30 (6): 1011-1019. doi: 10.1016/

j.cag.2006.08.002.

3. Boothroyd, G. 1988. “Estimate Costs at an Early Stage.”

American Machinist 132 (8): 54-57. doi: 10.1016/

j.ergon.2005.08.006.

4. Gatenby, D. A., and G. Foo. 1990. “Design for X (DFX): Key to Competitive, Profitable Products.” AT

& T Technical Journal 69 (3): 2-13. doi: 10.1002/

j.1538-7305.1990.tb00332.x.

5. Oh, J. Y., and W. Stuerzlinger. 2004. “A System for Desktop Conceptual 3D Design.” Virtual Reality 7 (3-4):

198-211. doi: 10.1007/s10055-004-0128-y.

6. Ullman, D. G. 2003. The Mechanical Design Process.

3

ed. New York: McGraw-Hill Higher Education.

7. Nagamachi, M. 1995. “Kansei Engineering: A New Er- gonomic Consumer-Oriented Technology for Product De- velopment.” International Journal of Industrial Ergo- nomics 15 (1): 3-11. doi: 10.1016/0169-8141(94)00052-5.

8. Kalyuga, S., P. Chandler, and J. Sweller. 1988. “Levels of Expertise and Instructional Design.” Human Factors 40 (1): 1-17. doi: 10.1518/001872098779480587.

9. Blomé, M., C. R. Johansson, and P. Odenrick. 2006.

“Visualization of Ergonomic Guidelines-A Comparison of Two Computer Aided Systems to Support Vehicle Design.” International Journal of Industrial Ergonomics 36 (6): 571-580. doi: 10.1016/j.ergon.2005.08.006.

10. Hidrio, C., and É. Jamet. 2008. “Learning from a Multi- media Explanation: A Comparison of Static Pictures and Animation.” In Understanding Multimedia Documents, edited by J. F. Rouet, R. Lowe, and W Schnotz, 103-118.

New York: Springer.

11. Höffler, T. N., and D. Leutner. 2007. “Instructional Animation Versus Static Pictures: A Meta-Analysis.”

Learning and Instruction 17 (6): 722-738. doi: 10.1016/

j.learninstruc.2007.09.013.

12. Höffler, T. N., and R. N. Schwartz. 2011. “Effects of

Pacing and Cognitive Style Across Dynamic and Non-

Dynamic Representations.” Computers & Education 57 (2): 1716-1726. doi: 10.1016/j.compedu.2011.03.012.

13. Crumpton, L. L., and E. L. Harden. 1997. “Using Virtual Reality as a Tool to Enhance Classroom Instruction.”

Computers & Industrial Engineering 33 (1-2): 217-220.

doi:10.1016/S0360-8352(97)00078-8.

14. Kuo, C. F., and M. J. J. Wang. 2007. “Interactive Virtual Evaluation of Interface Design.” Human Factors and Ergonomics in Manufacturing 17 (5): 485-495. doi:

10.1002/hfm.20088.

15. Dong, W., Z. Li, J. Yan, X. Wu, Y. Yang, and L. Zheng.

2008. “Adaptive Interaction in a 3D Product Structure Browsing System for Maintenance Training.” Human Factors and Ergonomics in Manufacturing 18 (1): 14-29.

doi: 10.1002/hfm.20093.

16. Lin, J. J. Y. 2008. “Research on Collaborative Concept Design Integrating the Application of Virtual Reality and DFMA.” In Proceedings of 12

International Con- ference on Computer Supported Cooperative Work in Design, Xian, China, 16-18 April 2008: 727-732.

Washington, DC: IEEE.

17. Bruno, F., and M. Muzzupappa. 2010. “Product Inter- face Design: A Participatory Approach Based on Virtual Reality.” International Journal of Human-Computer Studies 68 (5): 254-269. doi: 10.1016/j.ijhcs.2009.12.004.

18. Kim, C., C. Lee, M. R. Lehto, and M. H. Yun. 2011.

“Affective Evaluation of User Impressions Using Virtual Product Prototyping.” Human Factors and Ergonomics in Manufacturing & Service Industrie 21 (1): 1-13. doi:

10.1002/hfm.20210.

19. Noon, C., R. Zhang, E. Winer, J. Oliver, B. Gilmore, and J. Duncan. 2012. “A System for Rapid Creation

and Assessment of Conceptual Large Vehicle Designs Using Immersive Virtual Reality.” Computers in Industry 63 (5): 500-512. doi: 10.1016/j.compind.2012.02.003.

20. Melemez, K., G. Di Gironimo, G. Esposito, and A.

Lanzotti. 2013. “Concept Design in Virtual Reality of a Forestry Trailer Using a QFD-TRIZ Based Approach.”

Turkish Journal of Agriculture and Forestry 37 (6):

789-801. doi: 10.3906/tar-1302-29.

21. Li, H., T. Daugherty, and F. Biocca. 2001. “Character- istics of Virtual Experience in Electronic Commerce: A Protocol Analysis.” Journal of Interactive Marketing 15 (3): 13-30. doi: 10.1002/dir.1013.

22. Alcaide-Marzal, J., J. A. Diego-Más, S. Asensio- Cuesta, and B. Piqueras-Fiszman. 2013. “An Exploratory Study on the Use of Digital Sculpting in Conceptual Product Design.” Design Studies 34 (2): 264-284. doi:

10.1016/j.destud.2012.09.001.

23. Tiainen, T., A. Ellman, and T. Kaapu. 2014. “Virtual Prototypes Reveal More Development Ideas: Com- parison Between Customers' Evaluation of Virtual and Physical Prototypes: This Paper Argues that Virtual Prototypes are Better Than Physical Prototypes for Consumers-Involved Product Development.” Virtual and Physical Prototyping 9 (3): 169-180. doi: 10.1080/

17452759.2014.934573.

Manuscript Received: Sep. 03, 2015

First Revision Received: Sep. 11, 2015

Second Revision Received: Dec. 15, 2015

and Accepted: Jan. 06, 2016