Along with the rapid development of virtual reality (VR) technology and human-machine interface devices in recent years, haptic devices have been applied to many areas, such as entertainment, education, training, industry, manufacturing, telerobotics, art, and so on, to enhance the reality of virtual objects or provide force assistance, and thus to foster manipulative feeling or to improve task performance [24,57]. In human-machine systems, VR is often used to yield the user an illusion of reality and a feeling of “being there”
via real-time interaction with her/him through many different sensorial channels, such as vision, audition, and haptics [9–11]. Unlike that a single visual or auditory device can only serve one-way communication with the user, e.g., a display or speaker, a haptic device is bilateral for the user, such as force-feedback gloves or joystick, which enables the user can percept or manipulate the virtual or remote object via physical contact, including both tactile and kinesthetic senses [33,52,56,57].
According to our survey on previous works related to haptic applications in ma-nipulative tasks, basically, there are two major purposes of applying the haptic devices in human-machine systems: i) to emulate real objects or motions, such as telepresence, simulation systems for skill training, haptic display of virtual object, and ii) to provide assistance to the user by virtual objects for guidance or protection. To render the haptic effects, in the former the force information may be obtained from remote force sensors or simulated models, which may include geometric and physical properties, e.g., shape, hardness, or stickiness. We classify them into two categories below:
1. Sensor-based method: force-reflecting teleoperation [23,32]
2. Model-based method:
• Dedicated-function: simulation systems of calligraphy [36,61] and manual gearshift [18], skill training system for arc welding tasks [62]
• General-function: 3D haptic display [54,64], multi-functional virtual manipula-tion system for simulated tasks [26]
However, in the latter, the force information comes from the designer, which may be based on predefined environments, predictable user motion or environment, real-time sensed results, or user’s choice. We further classify them into four types below:
1. Predefined type: virtual fixtures (VF) [53] and virtual mechanism [28] for teler-obotic manipulation, geometrically-based VF in VR training environments [31,49, 50], anatomy-based constraints for surgery tasks [34], velocity-based guidance VF for telemanipulation [4,5]
2. Predicted type:
• User motion: HMM-based VF for path following and off-path motions [35], adaptive VF for teleoperation tasks [3]
• Environment: predictive haptic guidance for driving support [17], predicted-position VF for protection in robot-assisted surgery [20]
3. Sensor-based type: vision-based VF for microscale manipulation [8], laser-assisted for telerobotic system [60] or robotic surgery [51]
4. User-based type: virtual tools for robot manipulation [25]
According to the survey above, there are two important concepts, which are often used to enhance human performance in manipulative tasks: virtual fixtures and virtual mech-anism. In 1993, Rosenberg [53] first introduced the concept of virtual fixtures, which is an overlay of abstract perceptual information on a workspace to improve the human performance in a telerobotic task. For instance, a ruler can assist the user draw a line straighter and faster than freehand. Note that virtual fixtures may include not only hap-tic information but also visual or audio information. For the applications, virtual fixtures are often used to either serve for guidance, which lets the manipulated object move along desired paths, or to bound forbidden-regions, which protect it from hitting obstacles [4].
In 1995, Joly and Andriot [28] proposed the concept of virtual mechanism also for a telerobotic system. Virtual mechanism can confine both the master and slave arms indi-vidually, to move along a desired task space (e.g., a line or a circle), via virtual springs and dampers attached between them, in which the constrained forces are calculated by real-time simulation of this mechanical system.
Motivated by the concepts of virtual fixtures and virtual mechanism, in this disserta-tion, the virtual motion constraint is generated via the software, so that the force-reflection joystick, operated by the user, is confined to move within a limited workspace that cor-responds to task requirements or provides the user the real-time assistance. Therefore, the main purpose of this dissertation is to propose methods to design virtual motion con-straints, which can be applicable for well-known or uncertain environments, and apply them for simulation systems and robot manipulation.
First, for a well-known simulated task, we assembly virtual walls to construct the desired motion constraint and thus develop a multi-functional virtual manipulation sys-tem, based on a 2-DOF (degree-of-freedom) force-reflection joystick, which can emulate
various manipulative devices, e.g., a wrench or gearshift level. Differing from the previous works of dedicated simulation systems [18,36,61,62], which only can emulate a single de-vice for exact replication based on detailed analysis or dedicated hardware, we intend to let this virtual manipulative system be multi-functional via a systematic way to assemble different kinds of virtual walls, which can capture the main features of various kinds of manipulative devices, at the expense of exact replication. We also propose a pixel-based method, which can easily construct these motion constraints by using the graphic editing software and also maintain smooth force rendering between the walls. The experimental results show that this system can exhibit certain degree of resemblance and reality.
Second, for manipulating a multi-DOF robot manipulator in uncertain environments, also based on the concept of virtual motion constraint, we develop a set of virtual tools, e.g., a virtual ruler. These tools can provide the position or orientation assistance to the user, according to her/his call and setup on site, to govern the robot effectively. For instance, in a screw fastening task, the operator first manipulates the hex key, which is mounted on the robot manipulator, into a hex socket screw with a proper orientation.
Then, she/he calls the virtual tools to let this hex key maintain a fixed rotation axis and also move only along with this axis. Thus, with the help of virtual tools, the task becomes more easier than the case without them. Differing from the predicted or sensor-based methods for uncertain environments [3,8,17,20,35,51,60], when and what kind of virtual tools to use in our method completely depends on the user’s decision. We implement virtual tools, including a point, line, and plane, and provide methods for smooth force rendering during the transition between the use of two consecutive tools in guidance. In addition, a virtual spring (also a kind of motion constraint) is connected to the force-reflection joystick and the robot manipulator, so that the linkage between the user and
the robot can be established for better manipulation. For contact tasks, a virtual bumper attached on the robot manipulator for protection when interacting with stiff environments.
For demonstration, the system is employed for the tasks of contour following and screw fastening, both of which involve delicate and challenging maneuvers. According to these two different design concepts, we name them as “environment-oriented motion constraint”
and “user-oriented motion constraint,” respectively.
Among various kinds of haptic devices [24,56], we choose the force-reflection joystick to develop our manipulation systems, due to the merits of simplicity and generality.
Regarding the desirable features of force-reflection interfaces, Srinivasan and Basdogan [57] summarized three requirements: i) low back-drive inertia and friction, ii) ergonomics and comfort, and iii) matchable range, resolution, and bandwidth. And, according to [33,57], typical forces for humans used in exploration and manipulation are in the range of 5 to 15 N, and the spatial resolution of two points about 1 mm on the fingertip. For most of haptic systems, the force updating frequency of 1 kHz or more is demanded to provide a high quality of haptic rendering to the user [24,52,57].