4. EXPLORATORY RESULTS
4.4 Spatial-Temporal Patterns
From the analysis of bimanual actions, we saw two main kinds of spatial-temporal pattern of bimanual actions in assembly and adjustment stages. First kind is sequent placing/adjustment. It is similar to serial assembly in Guiad’s Kinematic Chain model. This kind of pattern emerges in placing part, adjusting posture and sculpturing by tools tasks. Two hands collaborate with each other step by step (Figure 4.13). Holding hand turned the doll and let the appropriate position to face to free hand.
Then free hand puts the part on the doll or adjusts the part. The participants focus on the particular position. So the process proceeds very fast.
Figure 4.13 Sequent placing/adjustment
29
Second kind is concurrent placing/adjustment. It is similar to the orthogonal assembly in Guiard’s Kinematic Chain model. This kind of pattern emerges in winding part and adjusting 3D posture tasks. Two hands move objects simultaneously. Holding hand control the orientation on z-axis and free hand control the position on x- and y-axis. The participants focus on every position of the part in whole process. So the process proceeds more precisely and slowly (Figure 4.14).
Figure 4.14 Concurrent placing/adjustment 4.5 Each Hand’s Role
Holding hand and free hand play different roles in different tasks. Holding hand is usually non-dominant hand and free hand is usually dominant hand. But they would interchange because orientation (Quote 3), symmetric (Quote 4) or personal habit. The roles of each hand are presented as follows.
Holding hand’s roles
(1) Assistance: Holding hand assists free hand to complete task. It fixes the object, so free hand could place object and add details on the doll. And it rotates the stick to let the appropriate position or parts to face to free hand, then free hand could place parts, adjust posture, and sculpt the doll conveniently.
(2) Coordination: When participants need to wind the part around the doll, two hands collaborate with each other simultaneously. Holding hand controls the orientation of the part on z-axis.
(3) Perception: Users could efficiently view the doll from different points of view by rotating and moving the stick. (Quote 7)
30
[U1] He said the stick could help him view the doll more conveniently and completely than without the
stick. (Quote 7)
Free hand’s roles
(1) Manipulation: Free hand directly manipulates the parts. It precisely places the part to the appropriate position. It move and rotate the part to adjust the posture.
(2) Coordination: As we noted above, free hand needs to collaborate with holding hand in more complicated task such as winding.
4.6 The Stick is a Jig
From video analysis, the stick is used to hold and orient the doll. Holding hand could rotate and move the doll with it. It could be considered a jig. By definition of Wikipedia, “jip is a type of tool used
to control the location and/or motion of another tool. A jig's primary purpose is to provide repeatability, accuracy, and interchangeability in the manufacturing of products.” It helps participants view and
manipulate when they pinch dough dolls. In addition, the roles of each hand are distributed clearly through the jig.(1) View: People view the doll more conveniently and completely with jigs compared to only by hands. Because jip is a long and thin stick, they could use both finger and wrist to rotate it. The range of rotation could be huge in small hand movement. (Quote 7) The jig could be a physical center line. It is a good reference for people checking the symmetry of doll. (Quote 8)
[U2] He says he could confirm where the center line of doll is by watching the stick. (Quote 8)
(2) Manipulation: In the 3D assembly, when people places parts or adjust parts, holding hand could keep distance from the doll. Holding hand could avoid interfering with the manipulation of free hand and rotate the doll in meanwhile. In addition, the jig makes the manipulation more facile.
People could let the jig between fingers and knead the dough. After the part is finished, they could place it on the doll efficiently.
31
4.7 Correlation between Feedbacks, Operation, and Goal
Figure 4.15 Correlation diagram.
To apply the pre-existing knowledge to mobile device interface design, the goal, interaction method, and feedback are the important issues. Since the jig is an important tool in pinching dough dolls like the mobile device held by hand. We focused these issues of the bimanual collaborations with the jig. There were four main goals of these bimanual interactions. We connected the operations and the related goals. Then the visual feedbacks that occur during these operations would be connected to related operations. (Figure 4.15) Four main goals and their correlate issues are described as follow.
(1) Let objects in appropriate position: In placing parts, holding hand would rotate the jig to assist free hand moving the parts. In winding parts, holding hand rotates the jig and the free hand moves the part simultaneously. The people could see the 3D rotation of doll and the movement of the part with hand.
(2) Let objects fixed: In placing parts and winding parts, free hand would press the part to fix it, and holding hand would fix the doll.
(3) Make particular posture: In adjusting posture, holding hand would rotate the jig to assist free hand adjusting the posture of part.
(4) Show appropriate parts of whole: In placing parts and adjusting posture, holding hand rotates the doll to let the appropriate position to face to free hand. For viewing, it also rotates the doll to display the appropriate face for people checking shape of the dolls.
4.8 Summary
In the process of pinching dough dolls, handcrafters need to prepare parts, to assemble parts, to adjust the posture, and to sculpt detail. In these stages, there are many different kinds of bimanual interaction which are similar to parallel, orthogonal, and serial assembly in KC model. Based on the analysis of bimanual interaction, we have investigated the spatial-temporal patterns, roles of each hand, and the usage of the stick.
32
When handicraftsman focus on a particular position, holding hand rotates the stick and let the doll face to free hand. Then free hand could place or adjust part conveniently. When they need to focus on every position during a more complicated manipulation, two hands would cooperate with each other seamlessly and simultaneously.
In these bimanual interactions, each hand’s roles are separated due to the stick. Holding hand play three key roles: assistance, coordination, and perception. Free hand play two key roles:
manipulation and coordination. Non-dominant hand’s and dominant hand’s role would interchange because the orientation and habit.
The long and thin stick is like a jig in pinching dough doll. This jig could help distribute roles to each hand. It could let the manipulation and view task more convenient and efficient.
33
5. DESIGN PATTERNS FOR BIMANUAL INTERACTION IN MOBILE DEVICE
Our intention is to allow two hands interacting naturally with mobile devices. The exploratory results showed smooth bimanual collaborations of moving, fabricating, winding, viewing, and adding detail in pinching dough dolls. Furthermore, before applying the pre-existing knowledge to interface design, the context of task should be considered (Carroll et al,. 1998). First, we explored the similar contexts between mobile device interaction and pinching dough dolls. The goal of tasks in each area would be considered. Second, through critical analysis of pinching dough dolls, roles of each hand, and benefits of the jig, we presented three design patterns for bimanual interaction in mobile devices.
5.1 Mapping between Physical and Virtual
Figure 5.1 Analysis of gesture designs in mobile devices.
In chapter 4, we have found out the main goals of bimanual interactions and the related manipulations and feedbacks in pinching dough dolls. In order to find appropriate context in mobile devices, we surveyed gesture designs that have similar goals in mobile phone first (Saffer, 2008;
http://www.youtube.com). According to the four main goals found in chapter 4, the gestures were connected to related goals. Each gesture also connected to the GUI (graphical user interface) effect caused by the gesture. Figure 5.1 showed the correlation between goals, gestures and GUI effect. But in mobile devices, users seldom adjust an object’s particular posture. For example, when users want to let objects in appropriate position, to let objects fixed and to show appropriate parts of whole, there are correlate designs.
Then, we made a mapping between the physical (pinching dough dolls) and virtual (mobile devices) (Figure 5.2).
34
Figure 5.2 Mapping between tasks in pinching dough dolls and mobile device interface.
35
Figure5.2 revealed a clear relationship between pinching dough dolls and manipulating mobile phone. People use different operation/gesture while pinching dough doll and using mobile phone, but they have the same goal while doing some tasks. In these tasks, we thought the pre-existing knowledge in pinching dough dolls could be mapping to mobile device interaction. Following are descriptions of three kinds of goal and related tasks.
(1) Let objects in appropriate position: In mobile device, when users use some functions such as relocation (Table 5.1) or category, they usually drag the object to the appropriate position no matter the position is on the screen or out of screen. If the position is out of screen, they drag the object near the side of screen, then background slide to next page. In pinching dough doll, if the target isn’t face to free hand, holding hand would rotate the doll and let the target position face to free hand. Then free hand place the object to target.
Table 5.1 Steps of relocation task in iPod/iPhone.
Steps Operation Goal
Switch to movable mode. Long press Let the icon movable Move the icon to the edge of the
screen.
Drag Let the icon place to
appropriate position
The screen slides to next one. Automatically Show the appropriate part of the whole.
Move the icon to appropriate position
Drag Let the icon place to
appropriate position Switch to regular mode Press the hardware
button
Fix the icon
(2) Let objects fixed: After users move objects, they need to fix them, they would tap or release the object in mobile device because the objects are be align or stuck. In pinching dough dolls, people need to press the object because of the gravity.
(3) Show appropriate parts of whole: When people want to view other parts of an object in mobile device such as different pages of menu or the track list of an album, they usually use slide on the screen to switch the pages. For example, when people rearrange icons, they often need to select appropriate area to place the icons (Table 5.2). In pinching dough dolls, they rotated the jig to view different faces of dolls.
36
Table 5.2 Steps of placing the icon to the appropriate panel in Android.
Steps Operation Goal
Let the icon that is in the menu move above the panel
Long press Let the icon movable.
Select the appropriate panel Slide Show the appropriate part of the whole.
Switch to regular mode Long press Fix the icon
The goals of steps in aforementioned task are similar to tasks in pinching dough dolls. People relocate objects or move objects from one place to another. Sometimes they need to move the object outside the screen or to another application. Current system would automatically switch to next page or another application. To aim at this kind of task, the exploration of pinching dough doll could be applied to build bimanual interaction.
5.2 Design Patterns for Bimanual Interaction in Mobile Devices
Design patterns are used to describe good practices. They are clear documents which help designers address similar questions quickly. They describe not only the solution but also the problem context (Borchers, 2001). After concerning the appropriate tasks, contexts, and limitations in mobile devices, following three design patters are built according to the results of exploratory study. They indicated the division of labor, gesture designs, and metaphor while designing bimanual interaction in mobile devices.
5.2.1 Pattern 1 3D Manipulation with Jig
This pattern describes the gesture design for rotating the 3D object with the jig. When system provides a 3D object for information visualization, user could rotate the jig to select different faces of the 3D object. The jig could be mobile device itself or be a graphic. When it is the mobile device itself, users tilt the device to rotate the objects (as shown in the figure of type 1 in Table 5.3). When it is a graphic on the screen, users could use finger slide on it to rotate the object (as shown in the figure of type 2 in Table 5.3). The goal of rotation is to select the appropriate face of the 3D object.
When users rotate the jig or tilt the mobile device, the 3D object would rotate to next face automatically. The motion graphic of rotation provide a feedback. The designer should notice that the manipulation of jig shouldn’t bother the manipulation of the other hand (Table 5.3).
37 Table 5.3 Pattern 1 3D Manipulation with Jig
What
3D objects could be rotated through manipulation of the jig. The jig could be graphic on the screen or the mobile device itself.
When
When system provides a 3D object for information visualization and each face of the 3D objects consists of different information, users need to select different faces for viewing or placing objects.
Interaction Suggestion
[Type 1] Mobile device is the Jig
There is a 3D object on the screen. Users tilt the device and the 3D object would automatically switch to next face. The direction of rotation is as same as the direction of tilt.
[Type 2] Graphic Jig
The virtual jig would under the 3D object. The finger lightly touches the jig and moves horizontally to rotate the object. When the object rotates more than a specific angle, it would automatically rotate to next face.
38 5.2.2 Pattern 2 Bimanual Relocation of Object
This pattern describes the layout and interaction for relocating an object with two hands, especially when users want to move the object outside the screen. There are two components in the relocation task: the background and movable items. The manipulations of object and the background could be distributed to free hand and holding hand. The background could be the faces of a 3D object, like the doll in pinching dough dolls. Holding hand could select different face of the 3D object by rotating it with the jig. The movable object could float above the 3D object (background). When the item floats up, it wouldn’t move with the object, and the object rotates behind it. When the item is attached on the face, it would move with the object. Free hand could move and position it. The visual effect should let users to feel clearly that the object is floated. This pattern is a natural interaction that mimics the physical world like moving the item to different position on the face or to different face of a 3D object such as pinching dough dolls (Table 5.4).
Table 5.4 Pattern 2 Bimanual Relocation of Object What
The object and the background could be separately manipulated by each hand. Two hands collaborate to relocate objects.
When
Users need to relocate the object, especially moving it to outside the visible viewing area.
Interaction Suggestion
The system could provide a spatial metaphor that the items are placed on a 3D object such as a cube. User could select different faces of the 3D object by rotating it. The item could be floated above the 3D object. When the item floats up, it wouldn’t move with the cube. The cube would rotate behind it. Holding hand could control the 3D object by the jig. Free hand could move the item, float it up and attach it to the face.
39 5.2.3 Pattern 3 Bimanual Object-Placing
This pattern describes how to place items from one place to another with two hands such as moving the photos to different albums. The area of destination and area of the items are displayed separately on the same screen. Holding hand could select the sub-areas of the destination. Free hand could select and move the item in the meanwhile. This interaction is similar to that subjects kneaded the dough on their palm and then placed on the particular position of the doll (Table 5.5).
Table 5.5 Pattern 3 Bimanual Object-Placing What
Placing items from one place to another with two hands on the same screen.
When
When users need to move items from one place to another place such as moving the photos to different albums, they need to select both the items and the appropriate sub-area in the destination.
Interaction Suggestion
Two hands could select the items and sub-areas separately. Free hand selects the item and moves it. The destination could be represented as a 3D object. Holding hand could select the face of the 3D object by tilting the device.
For example, the area where items originally are is displayed in the lower area of the screen.
The destination, the cube, is displayed in the upper area. Free hand’s finger slides to select the items and drag the item to appropriate face. Holding hand tilts the mobile device to select between different face of the cube.
40
Based on results of exploratory study, following are two concerns for building bimanual interaction in mobile device.
(1) Two hand’s roles must be interchangeable.
From the exploratory study, although all subjects are right-handed, and they usually used left hand to hold the stick and right hand to pinch dough, two hands would exchange their roles because of orientation or personal habit. The interface design must let two hands easily exchange their roles.
No matter which hand they use to hold device, they could complete the task smoothly.
(2) System should allow two hands manipulate simultaneously.
Subjects in our study always move the part and rotate the stick in meanwhile especially when they wind part around the doll. There isn’t an obvious division between two hands’ action. System should allow two hands to manipulate the jig and the object simultaneously.
5.3 Summary
In mobile device, the goals of some task are similar to the bimanual interaction with the jig in pinching dough dolls such as relocation and categories. Different subtasks in these tasks could distribute to each hand. In these contexts, through the analysis of exploratory study; we presented three design patterns for bimanual interaction in mobile devices. These patterns indicate natural ways for viewing the 3D objects, relocating objects, and placing objects with two hands. They described the problem, the solution, division of labor, usage of metaphor, and examples.
41
6. USER EVALUATION
This chapter described the process and result of user evaluations for the design guidelines presented in previous chapter. The prototype design and process of user evaluation was described in Section 6.1. In Section6.2, data of questionnaire and interview would be analyzed. Last, concerns of bimanual interaction in mobile device and the modification of design patters are described in Section 6.3.
6.1 Experiment
The experiment had two parts. First was the usability test. We built three prototypes according to the design patterns in chapter 5. Subjects provided their subjective feeling about easy-to-use, effectiveness, efficiency, joyful, and intuitiveness of prototype through questionnaire. The second part is semi structured interview. Following is the description of the prototype, and then the whole
The experiment had two parts. First was the usability test. We built three prototypes according to the design patterns in chapter 5. Subjects provided their subjective feeling about easy-to-use, effectiveness, efficiency, joyful, and intuitiveness of prototype through questionnaire. The second part is semi structured interview. Following is the description of the prototype, and then the whole