In-cell Optical-based

The in-cell optical-based 3D virtual touch system is the extension of the 2D in-cell multi-touch system. The continuous space from 2D surface to 3D virtual touch can be achieved without hybrid other mechanisms since they share the similar structure which was firstly proposed by W.D. Boer et al at 2003 [26]. Meanwhile, objects can be sensed under similar working principles; a hovering object can be detected either by the reflected light which is emitted by infrared LEDs in the LCD’s backlight, or by the object direct illumination on the display. Several techniques were proposed to determine the 3D coordinate (x, y, and z) and/or orientation (θ, and ) information.

2.5.1 ThinSight

The idea of extending 2D multi-touch to 3D interaction in an in-cell optical-based system was first proposed by S. Hodges et al in 2007 [15]. A hardware structure with infrared sensors and emitters integrated into a thin form-factor display was suggested to detect multiple fingers placed on or infrared-emissive objects near the display surface, as shown in Fig. 27 (a).

Fig. 27. (a) Basic construction of ThinSight, and (b) applications in ThinSight

24

display including 2D finger multi-touch, data transformation between a mobile device and the display, and 3D gesture interaction through a device which can cast a beam of infrared light onto the display. [15]

The layer of diffusing film between the LCD itself and the brightness enhancing film is removed because it cause too much attenuation of the infrared signal, especially for passive objects that reflect the light emitted from the display. However, the removal of diffuser has a detrimental effect on brightness and viewing angle of the LCD panel. Besides, it is not able to provide depth information. Still, based on the structure, a number of promising applications, such as data transformation, and 3D gesture interaction through an IR emitting device are proposed, as shown in Fig. 27 (b).

2.5.2 Sensible Backlight

A multi-touch LCD display architecture with hover sensing capability was proposed by K. Yi et al in 2010 [27]. Instead of changing an LCD manufacturing process to insert photo sensors in cell, the system proposes a sensible backlight where a backlight unit is integrated with an IR sensor array, as shown in Fig. 28. IR light sources for touch and hover detection are positioned on the bezels of the display. The 2D multi-touch is achieved based on frustrated total internal reflection (FTIR). For recognizing simple hovering objects, side IR illuminators emit light with tilt angles.

Hence the approximate hovering positions can be obtained by sensing the reflected light by the IR sensors sensible backlight.

25

Fig. 28. (a) Working principle and optical structure of the sensible backlight system, and (b) layout of backlight with RGB LEDs integrating with IR photo transmitters.

[27]

Fig. 29. (a) Sensor image of touch, (b) sensor image of hover, (c) extracted touch points, and (d) extracted hover points. [27]

The 2D multi-touch and hover positions can be obtained by utilizing the sensible backlight system. However, the depth information cannot be found, as shown in Fig.

29 (d). The approximate hover position, which is determined by the center of mess of each captured image, outputs a 2D coordinate on the surface. Moreover, a thicker border is needed for illuminated infrared light on the hover objects, and the panel size is limited by the intensity of side illuminated light. Therefore, the system is more

26

applicable on middle size displays with near-distance interaction.

2.5.3 Directional Image Sensor

A LCD system with integrated 3D input device was proposed by C. Brown et al in 2010, [28] and the concept is illustrated in Fig. 30 (a). The 3D input function is successfully achieved by employing directional image sensors integrated onto the TFT substrate. The directionality of the image sensor is created by an upper light shield formed in a second additional metallization layer, as depicted in Fig. 30 (b), to allow only incident light within a specific angle being detected. A set of four orthogonal direction sensors is able to generate four unique directional sub-images from the detected light. Sub-images are then processed to extract planar coordinates (x,y) corresponding to the location of the peak output signal, as shown in Fig. 31.

Finally, by examining the relative displacement (d) of the object’s 2D coordinate in each directional sub-image, the depth value (z) can be calculated by Eq. (1).

d = 1

2 𝑧 ∙ cos 𝜃

Eq. (1)

Fig. 30. (a) Concept of a 3D input device using a directional image sensor, and (b) structure of thin-film lateral diode and light shielding layers to create a directional field-of-view. [28]

27

Fig. 31. One dimensional representation of the response of two directional sub-pixels. [28]

However, due to the construction of the panel, limited field-of-view of the directional sensor, and the insufficient sensitivity of the image sensor, detected z-axis exhibits a limited linear response from 0 to 20 mm. It is not enough for near field 3D interaction where 50 mm depth can be perceive in an auto-stereoscopic display.

Meanwhile, the shielding layer above the photo sensor significantly reduced aperture ratio which makes the system hard to be implemented on mobile devices because of the considerable power consumption.

2.5.4 Color-filter-based Sensing

A 3D multi-interactive system achieved by color filter based sensing was proposed by H.Y. Tung, et al [29] under in-cell optical based structure where photo sensors are embedded on a TFT substrate in a LCD. As illustrated in Fig. 32, multi-wavelength of light sources, red, green, and blue are utilized as interaction illuminators. By using a color filter as a band-pass filter, different users can be identified by extracting the sensor outputs under accordant color filters. After separating light sources, center and radius of each circle are calculated and used to determine the 2D coordinate (x and y) and height (z) respectively. Therefore, 3D multi-touch interaction with user recognition can be achieved.

28

Fig. 32. System structure of color filter based sensing.

However, due to the usage of visible light sources, the display quality is degraded. The limited number of three users can be achieved due to the limited kinds of color filter in a display. Moreover, the imperfection of color filter would result in light leakage where light spots are too close and overlapping happens, as shown in Fig.

33. Nevertheless, a complex image processing, Hough transform, needs to be applied for circle detection which results in heavy computation.

Fig. 33. A detected image under red color filter. Undesired blue light penetrates a red color filter and overlapping of red and blue spots occurs.

2.5.5 Multi-mark Based

A 3D multi-mark interactive system was proposed by C.C. Chao, et al [30] to establish an interface between multiple users and 3D images. Based on the in-cell optical-sensor structure, illuminators such as IR light pens covering with designed marks, as shown in Fig. 34, is utilized to provide significant features. The display quality can be maintained by utilizing IR LEDs. By the proposed algorithm, different

29

users and their 3D coordinate (x, y, and z) can be identified. Meanwhile, the overlay of a circle with one of other marks, T or X, can be distinguished for up to 40%

overlapping. Finally, the proposed algorithm was successfully demonstrated on a 4”

panel.

Fig. 34. Designed light marks of solid T shape, X shape, and circle.

However, the limited of 3 users can be achieved due to the specific designed mark. Meanwhile, the complex overlapping cases, as illustrated in Fig. 35, dramatically increase the loading on a circuit, which also increases the cost and power consumption. Moreover, due to the similar characteristic of T shape and X shape, the overlapping of T and X cannot be identified in the system where only two users overlapping with one of them must be a circle can be deal with.

Fig. 35. Complex overlapping conditions need to be processed case by case which results in heavy computation requirement.

30