Preface

Chapter 1 Introduction Introduction

1.1 Preface

Recent years, the battle field of technology competition has moved from the hardware into the software. The software competition, to be more precise, refers to pursuit of a friendly user interface and a better user experience. As shown in Fig. 1, in the early 1870s, C. Sholes invented the QWERTY keyboard [1] as an input device for a computer. Until 1964, the first prototype computer mouse [2] was made to use with a graphical user interface. In the next year, the first two-dimensional (2D) touch screen was developed. However, the touch technology does not dramatically change the user habits until the bump up of the smart phone. In the beginning of 21^st century, the wide adoption of smart phones and tablets have accelerated a user interface transformation and paved the way to multi-touch technologies. Multi-touch technologies started to play an important role in our lives. It became so unexpendable in today’s world since Apple established multi-touch as a “must-have” technology.

The result is that people of all ages expect every display to be touchable with multiple fingers.

2

Fig. 1. Trend of interfaces and 3D displays technologies.

1.2 2D Multi-touch Technologies

Multi-touch technology is defined as the ability to recognize two or more contact points at the same time. [3] In 2004, the first commercial product JazzMutant’s Lemur music [4], as shown in Fig. 2, brings multi-touch technology into our lives.

Fig. 2. First 2D touch product with multi-touch user interface - JazzMutant’s Lemur music.[4]

At present, the mainstream of 2D multi-touch technologies are projected capacitive, analog resistive, and camera-based optical. These technologies increase the flexibility between human and machine by allowing a user to simultaneously control rotation, scaling, and translations (RST) form multi-touch gestures [5]. More

3

intuitive interactions and user interfaces have been achieved due to multi-touch technologies. In the following paragraphs, we’ll briefly describe these technologies.

Projected Capacitive: In the projected capacitive sensing system, there is a capacitor at every intersection of each row and each column. A voltage is applied to an X-Y grid, which is an electrode consisting of a matric of drive lines and sense lines, to form a uniform electrostatic field. As a finger or conductive stylus touches or close enough to the surface of the screen, it renders a distortion in the local electrostatic field, as shown in Fig. 3. The capacitance change at every individual point on the grid can be measured to accurately determine the touch location, as illustrated in Fig. 4. A single drive line is excited with an ac signal. The capacitance at each intersection between the drive line and each sense line is measured simultaneously. Next, multiplexer outputs the measured values and next being converted into a digital data stream by an A/D converter. Digital signal processing (DSP) is thus employed to interpret the data stream to 2D (x and y) coordinate of the touch location on screen surface.

Fig. 3. Principle of capacitive sensing: when a voltage apply, (a) there exist a capacitance between X electrode and Y electrode. (b) once a finger touch the surface, there is a distortion in electrostatic field and the mutual capacitance is reduced.

4

Fig. 4. Working principle of position determination in projective capacitive sensing system. [6]

Analogue Resistive: An analogue resistive touchscreen panel is composed of several layers, as shown in Fig. 5. The sensing layer is constructed of two sheets of material separated slightly by spacers: a sheet of glass providing a stable bottom layer and a sheet of Polyethylene (PET) as a flexible toy layer. The two sheets are coated with a resistive material, usually a metal compound Indium Tin Oxide (ITO) and separated by an air gap or microdots. When an object, such as a finger, presses down on a point on the outer surface, the two metallic layers connected at the point. This causes a change in the electrical current, which is registered as a touch event and sent to the controller for processing. Hence the position of the touch on the surface can be measured.

Fig. 5. Illustration of a resistive touchscreen. [6]

5

Optical: In conventional optical touch technology [7], an array of infrared (IR) LEDs were allocated on two adjacent bezel edges of a display with photo sensors placed on the two opposite bezel edges to determine a touch event. The photo sensor outputs can be used to locate a touch-point coordinate, as shown in Fig. 6.

Fig. 6. A schematic representation of conventional optical touch technology.

Although the traditional type of optical touch has been hampered by two factors, the relatively high cost compared to competing touch technologies and the issue of performance in bright ambient light, certain features of optical touch remain desirable and represent attributes of the ideal touch screen. For examples, the option to eliminate the glass or plastic overlay that maintain the display quality, the digital nature of the sensor output, no direct impact of a touch object, and multi-touch implementation are the promising features in optical-touch technology.

There are several new types of optical-touch systems that is able to detect multiple touches, for instance, vision-based, camera-based, and LCD in-cell optical touch.

6

Vision-based optical touch systems [8] employ one or more IR imaging cameras

to capture the image of the entire screen, which usually means that the camera must be located a significant distance away from the screen. Therefore, most vision-based touch systems, like frustrated total internal reflection (FTIR) [9][10], diffuse illumination (DI) [11], and diffused surface illumination (DSI) [12] systems as illustrated in Fig. 7 to Fig. 9, are implemented with the detecting cameras located behind a projection-screen surface and process the captured images to determine the 2D coordinates of touching objects.

Fig. 7. General set-up of a Frustrated Total Internal Reflection (FTIR) system which is based on optical total internal reflection within an interactive surface; as a user touches the screen, the light escapes and is reflected at the finger’s point of contact, which is then detected by an IR camera at the back of the pane. [8]

7

Fig. 8. General set-up of a Diffuse Illumination (DI) system which is similar to that for FTIR but with IR lighting placed behind the projection surface; moreover, DI is capable of tracking and identification of objects by using their shape or fiducially printed on their bottom surfaces. [8]

Fig. 9. General set-up of a Diffused Surface Illumination (DSI) system which utilizes small number of (two or three) infrared illuminators and evenly distributed the light across the screen surface. [8]

For a vision-based optical touch system, 2D multi-touch can be successfully achieved. However, the large system volume is essential. Therefore, a camera-based optical touch technology has been developed to reduce the size of the system.

8

Camera-based touch technology utilizes at least two line-scanning cameras, as

illustrated in Fig. 10, located at adjacent corners of a display. The light source is usually one or two IR LEDs that are integrated into each camera assembly. The light is emitted in a plane across the surface of the screen and is reflected back to the cameras by retro-reflecting strips located along three edges of the screen.

Fig. 10. Schematic of a camera-based optical touch technology. [13]

However, a thick bezel must be fabricated due to the illumination of IR light upon display surface. Hence, LCD in-cell optical touch is proposed to maintain the virtue of thin form factor in a display.

LCD in-cell optical touch, also called “in-cell light-sensing,” establish by

integrating light sensing element (photodiode or photo transistor) into some or all of an LCD’s pixels, as illustrated in Fig. 11, which allows the display to act as a large array photosensor. The photo sensors receive light by either sensing the shadow of touching objects causes by blocking the ambient light [14] , as shown in Fig. 12 or the reflected light from infrared emitters embedded in the LCD’s backlight [15], as shown in Fig. 13; a multi-touch controller samples each photosensor and calculates the X-Y coordinates of touching objects.

9

Fig. 11. General cross-section of LCD in-cell optical touch panel. [14]

(a) (b)

Fig. 12. 2D in-cell light-sensing can be achieved by (a) sensing the shadow of touching objects caused by blocking the regional ambient light where (b) the objects’

positions reveal lower gray level [14].

(a) (b)

Fig. 13. (a) The prototype of capturing the reflection of IR backlight is employed in 2D in-cell light-sensing where (b) the objects’ positions exhibit higher gray level. [15]

10