Time Delay Compensation

In this section we present an advanced control technique, time-delay compensation, which deals with a common problem in process control, namely, the occurrence of significant time delays. From a frequency response perspective, a time delay adds phase lag to the feedback loop, which adversely affects closed-loop stability. Therefore, the controller gain must be reduced below the value that could be used if no time delay were present, and the response of the closed-loop system will be inefficient compared to the control loop with no time delay.

In order to improve the performance of system with time delays, special control

strategies have been developed that provide significant time delay compensation. The Smith predictor technique is the best strategy that is known. A block diagram of the smith predictor controller structure is shown in Figure 2.8.

Figure 2.8 Block diagram of Smith predictor.

After some computation, the closed loop set point transfer function is:

*

where G_T(S)=e^−θsis estimation of process delay. By contrast, for conventional feedback control

Compared with conventional feedback control, the Smith predictor has theoretical

advantages of eliminating the time delay from the characteristic the time delay from the characteristic equation. But it has a serious disadvantage, the advantage is lost if the process model is inaccurate. Fortunately, the delay in camera system is usually constant and not large, which is caused by processor computation and CCD/CMOS capture. Another disadvantage of the Smith predictor approach is that it is model-based; that is a dynamic model of the process is required.

Chapter 3

Principles of Auto-Focusing System

This chapter describes the auto-focusing process from the perspectives of both digital image processing and signal processing. Auto-focus is an important pre-process of a digital camera system, which adjusts the focus position. It receives and processes the raw data obtained from the image sensors. The auto-focusing system, unlike the other pre-processes, does not directly alter the raw data, but it does provide feedback. In other words, an auto-focusing system cannot be an instant or real-time system. It needs a little bit of time to complete the auto-focusing process. How to extract the focus value from the image data and how to reduce the time of focusing process are the critical issues in the development of a robust and practical auto-focusing algorithm.

Figure 3.1 Block diagram of an auto-focusing system

Figure 3.1 shows a block diagram of an auto-focusing system, which is simplifies from the model of a digital camera system. The major parts of the auto-focusing algorithm are the focus value metric and the search algorithm. We also introduce the window selection, which is

3.1 Focus value and Focus curve

The first step in the development of a robust auto-focusing system is to determine a good focus value metric. This step is classified as feature extraction, which compresses a large amount of data into a simple form with sufficient accuracy. So the focus value is defined as the focus measurement of an image data. A good focus value model should have the following properties: First, its process should be independent of the received data and its measurement is universal for all types of image structures. Second, it should be a decreasing function with respect to the true focus of the target. If the image is sharper, the focus value gets higher.

Third, it can tolerate disturbance and noise. Even though the camera system is disturbed by noise and delay, its variation is small and predictable. In theory, a blurred image is the output of a focused image convolved with a band-limited low-pass filter. Therefore, a simple focus value model is detecting the high-frequency components of an image.

Figure 3.2 A general focus curve.

Figure 3.2 shows an example of focus curve, which plots the focus position against the focus value derived using a Sobel operator. A focus curve, which shows the focus values at different positions (locations), is a result of several factors. Its shape depends on the image, the optical system, the lighting condition, and the metrics (operator). The general shape of many published focus curves is similar. They often have a peak with a sharp tip and two steep slopes. The tip of the focus curve shows the optimal focus position of the scene. A close-up of such a focus curve reveals small magnitude disturbance that are due to thermal noises. Many research and published papers assume that focus curve is the basis in designing and evaluating an auto-focusing system. An auto-focusing system with stable focus curves indicates its system design is more robust, because the stable focus curves shows that exceptional condition is very rare in this auto-focusing system. Although many commercial devices have good auto-focusing systems, there still lacks a high performance focus value model in the literature. Certain types of frequency transforms or image operators are widely used as the focus value model as describe below.

3.1.1 Sobel Operator (First Derivative)

The Sobel operator is a first-derivative operator. First derivatives in image processing are implemented using the magnitude of the gradient. This is also an image enhancement in the spatial domain, which convolves a designed filter mask with the original image. Masks of even size are awkward to implement. The smallest filter mask is a mask of size 3*3.

Figure 3.3 Sobel masks and equations.

The mask of the sobel operators/filters shown in Figure 3.3 can be used for the edge

enhancement. In Figure 3.3, A is the original image, and G_x and G_y are derivatives along the horizontal and the vertical axes. The resulting gradient approximation G is the square root of Gx and Gy. A weight value of 2 at the center is giving more importance to the center point.

Note that the coefficients in the masks shown in Figure. 3.3 sum to 0, indicating that they would give a response of 0 in an area of constant level, as expected of a derivative operator.

Mathematically, the Sobel operator is not a linear operator because of the square root calculation. Figure 3.4 illustrates the computation of the Sobel operators on the image of Lena.

Figure 3.4 Original image and Sobel gradient.

3.1.2 Laplacian of Gaussian (Second Derivative)

The Laplacian of Gaussian filter (LoG) is a convolution of a 2^nd derivative filter. The LoG value of an image region of rapid intensity is often used as an edge detector. This approach is a specific spatial filter with impulse response related to Gaussian signal shape.

Figure 3.5 Laplacian masks.

Because the Laplacian kernels are approximating a second derivative metrics on the image, they are very sensitive to the noise. To depress noise, the image often smoothed by a Gaussian low-pass filter that reduces the high frequency noise components prior to the differentiation step Laplacian filter. Therefore, since the convolution operation is associative, we can convolve the Gaussian smoothing filter with the Laplacian filter first of all, the LoG filter, and then convolve this hybrid filter with the image to achieve the required result. The function of Laplacian of Gaussian is

2 more flexibility that it can be decomposed into separate linear filters. Employing the functions

smaller and the function can be easier to implement. Similar to the LoG, difference of Gaussian (DoG) approximates the LoG filter with combination of two different Gaussian filters. The result of the DoG filter is similar to that of the LoG filter, but it has the advantage of computation complexity.

Figureure 3.7 illustrates the output of the LoG filter on the image of Lena compared to the original image. Note that the noise components are much higher than Figure 3.4 shows, which is the serious side effect of 2^nd derivative based technique.

Figure 3.7 Original image and LoG result.

3.1.3 Wavelet Transform (Frequency Domain)

Wavelet transform [14] is a spatial decomposition that is done along the horizontal and the vertical directions. Wavelet transform also contains frequency information in each sub-band. Therefore, the transform coefficients reflect the energy distribution of the source image in both space and frequency domains. Wavelet transforms are broadly classified into the discrete wavelet transform (DWT) and the continuous wavelet transform (CWT) ,and we briefly describe the DWT.

Figure 3.8 A 3 level filter bank. [3]

The decomposition of wavelet transform is repeated to further increase the frequency resolution and the wavelet coefficients are produced by the high-pass and low-pass filters and then down-sampled. This is represented as a binary tree with nodes representing a sub-space with a different time-frequency localization. Figure 3.8 [3] shows a 3-level filter bank The same process can generate an N-level DWT decomposition.

Figure 3.9 3 level decomposition in spatial and frequency domain. [3]

Figure 3.9 [3] shows a 3 level wavelet decomposition. Because the DWT’s scalable kernels are used as lowpass and highpass filters, most Fourier-based filtering techniques have an equivalent wavelet domain approach. So the wavelet transform benefits greatly from the filtering techniques that reduces noise effect. Figure 3.10 illustrates the result of 2 level wavelet transform on the image of Lena. The high frequency bands may be use used for edge detection and thus representing focus value.

Figure 3.10 Original image and its wavelet transform.

3.2 Search Algorithms

In section 3.1, the definitions and usage of the focus value and the focus curve are introduced. It seems that an auto-focusing process is quite simple. It simply finds out the global maximum points of a focus curve. The only thing to do is to choose an efficient search or sort algorithm and implement it. However, a focus curve can only be acquired using offline simulation or a full scan operation. Typically, the focus curve is used to train the auto-focusing system in advance. An apparatus only receives or detects a single focus value at a time, not the entire focus curve at one time. Therefore, we need a search algorithm that identifies the best focus position with one by one sequential focus values.

Figure 3.11 Ideal output response of an auto-focusing system.

Before introducing of practical algorithms, let us consider a fundamental problem first, the ideal system response of an auto-focusing system. What the auto-focusing system does is adjusting the focus position to the location where the image looks sharp. It is reasonable to conclude that the best response of an auto-focusing system is near a step function response, as shown in Figure 3.11. But this is impossible for a practical auto-focusing system because of the following reasons.

1. An auto-focusing system is a feedback control system. A feedback control system cannot produce an instantaneous response without processing delay. There must be a system delay that tilts the slope of response curve.

2. The input is not the correct focus position. The input of an auto-focusing system is not the correct focus position but the focus values, which are clued of the correct focus position indirectly. Since the input is indirectly, it is hard to design a linear auto-focusing system to match the optimal response.

3. There are always delays and noise in a practical system, which causes the system response

imperfect.

Figure 3.12 A general output response of an auto-focusing system.

Figure 3.12 contains a general model of output response, plots of focus position versus time, for a practical auto-focusing system. The solid line, a step function response, indicates the ideal auto-focusing result, and the dot line, the general output response, shows the various stages in the search algorithm. The dot line state 1 and the dot line state2 are the main search process. The state 1 limited by the speed of the system response is a process of moving current focus position quickly to the near optimal position or, in other words, to the position with a high focus value. The state 2 is a process of identifying the position with the maximum focus value, the optimal focus position. Usually, the system goes beyond the focus position due to delay and needs to come back. The state 3 is a process of re-adjusting the focus position from the overshoot position to the optimal focus position found at state 2. The state 4 is a process of locking the current focus position and detecting whether the image object is changing its location and then a new auto-focusing process needs to execute again. Compare to the ideal step function response that contains only two line segments or states, state 2 and state 3 represent the imperfection of a practical auto-focusing system. The state 2 and state 3, also called over-focus, is one main indicator to judge the efficiency of a search algorithm.

Therefore, the correctness (for the given data) or the performance (in real environment)

determines whether a search algorithm is useful or not. A robust auto-focusing algorithm is better than a fast one, although a fast algorithm is more and more desirable for today’s digital camera system.

3.2.1 Full Search algorithm

The full search is the simplest and most robust search algorithm. Scan all the focus positions from far to near and choose the position with the maximum focus value for the correct solution. Full search algorithm provides a global maximum focus value, even though the system has serious noise and with a disturbed focus curve.

Figure 3.13 Output response of the full search algorithm. The best focus position is 14.

The full search algorithm is very robust but inefficient in time. The full search algorithm has a serious over-focus, the long line segments of state 2 and state 3. It is also difficult to implement a full search algorithm for the current camera apparatus because of the large

amounts of data. For example, there are 33792 focus positions in the SONY EVI-D100 camera. The modified full search algorithms, such as fabonacci search, reduce the number of search points and accelerate search process and then the output response has a steeper slope.

The full search algorithm is not a mainstream in the existing auto-focusing system, but it provides a reference for solving difficult cases.

3.3.2 Climbing Search Algorithm

The climbing search algorithm [18] is a fast search algorithm with an adaptive step size.

This algorithm is developed based on the shape of the mountain-like focus curve, and finding out the optimal focus position at the maximum focus value is similar to climbing to the summit of a mountain. The climbing search algorithm reduces the necessary search steps to increase search efficiency that but suffers from serious noise and with a disturbed focus curve.

Figure 3.14 The four states of the climbing search algorithm.

A basic climbing search algorithm contains four stages [5]: out-of-focus, nearly focused,

over-focused, and in-focus. Figure 3.14 shows each stage on the focus curve. The out-of-focus stage means the current focus position is far away from the optimal focus position. The nearly focused stage means the focus position is close to the optimal focus position. The over-focused stage means the focus position goes beyond the optimal position and becomes less focused. The in-focus case means the focus position is esteemed as the optimal focus position and the auto-focusing process terminates. Figure3.15 shows the state diagram model of the mountain climbing algorithm, which is widely adopted in electronic circuit design.

Figure 3.15 The state diagram of the climbing algorithm.

In order to identify the four stages, especially the out-of-focus and the nearly focused cases, we need a measurement typically uses the differences between previous and current focus values or equivalently calculates the slope of focus curve between two focus positions.

A positive value indicates the out-of-focus state. When the value gradually moves to 1, it indicates the nearly focused state. A negative value, s the current focus value is lower than the previous focus value, indicates the out-of-focus value. The accuracy of this measurement correlates highly to the performance of the focus value. In other words, the performance of a mountain climbing algorithm depends on the focus value/curve and may be disturbed or corrupted by serious noise and delay.

Figure 3.16 Output response of the mountain climbing algorithm. The best focus position is 14.

3.3 Window Selection

Window selection [4] is a common additional function, what is a spatial domain enhancement method in an auto-focusing system. It helps the auto-focusing system to locate the focusing target and improves the curve shape. Window selection has simple computation but powerful results.

Figure 3.17 The original screen and the 16 focus windows.

Figure 3.17 shows the 16 focus windows model. We divide the original screen into 16 rectangles. The number of windows depends on the design of the auto-focusing system. Fewer windows require fewer memories but also lose some spatial information. Figure 3.18 shows the image “Scarecrow” and the Sobel operator result. It clearly shows that the edge feature (focus cue) is not uniformly distributed but clusters at certain regions. How to locate these hot regions?

Figure 3.18 The picture and its edge feature. (focus cue)

Of course, the best solution is to identify the foreground or the targets, such as faces or people, and filter out the background that may confuse the auto-focusing system. This approach is an object recognition problem and still a challenge to the real time hardware implementation. Hence, simple window selection is a feasible solution from the viewpoint of hardware design. A good window selection adaptively adjusts the weightings of each window to locate the hot regions.

Figure 3.19 The focus curves of each window.

Figure 3.19 shows 16 different focus curves associated with 16 windows. It is apparent that the focus values of different windows behave differently at the same focus position. The overall focus value is a weighted sum of these window values.

∑

= w(i)fv(i)

wfv (1)

The main purpose of window selection is to select the window weighting coefficient for each window.

In general, the center windows have higher weights because most people put their attention on the center of an image. Another approach is that the window with the maximum focus value has the highest weight or chooses it as the focus window. The overall focus value comes from the focus window only.

Figure 3.20 The normalized focus curve comparison between no window selection (solid line) and maximum value selection (dot line).

Figure 3.20 shows the normalized focus value comparison between no window selection (solid line) and maximum value selection (dot line) for scarecrow using sobel operator. The dot line expresses the maximum value selection, which has shaper curve.

3.4 Conclusion

Figure 3.21 Flowchart of an auto-focusing algorithm for still image.

Figure 3.21 shows flowchart of an auto-focusing system for still image. The auto-focusing process begins when the start signal is received and the system gets the focus value with the metrics and window the selection method setting in advance. The auto-focusing will run the search loop continuously until the search algorithm finds the optimal focus position. Thus the system terminates the auto-focusing process and locks the focus position.

Although our purpose is to design a dynamic auto-focusing algorithm, there are parts of algorithms working well no matter for still image or video, such as the focus value metrics and the window selection method. We choose the Sobel operator as our focus value metrics

because it is well known and is less sensitive to noise. We also divide our focus window into 16 rectangles and adapt the maximum value selection to enhance our focus curve. The only

because it is well known and is less sensitive to noise. We also divide our focus window into 16 rectangles and adapt the maximum value selection to enhance our focus curve. The only