LMS algorithm

We want to find out the step-size which can minimize each tap coefficient error variance i.e., MSE for each iteration step. Hence, we use step-size diagonal matrix

Substitute Eq.(3.5) into Eq.(3.3) to obtain ˆ ( 1) We may now express the correlation matrix of the coefficient error as follows:

2

Eq.(3.7) disappear because of the independent and zero mean assumptions of the noise ( )z n .

We assume the step size is small, so that the LMS filter acts as a low-pass filter with a low cutoff frequency [1]. With this assumption, the variation of the coefficient error vector ( )v n with time is slow compared with that of the input vector ( )x n . In

The Gaussian assumption (see in the Appendix) is used next to simplify the third term in Eq.(3.9), then we can obtain

Eq.(3.11) is the general iteration form of tap coefficient error variance. If we assume

tap coefficient error variance are identical i.e., g n₀( )= =g_L−1( )n , then the fourth

term of Eq.(3.11) will become Lu n²( )σ_x⁴g n( ) and Eq.(3.11) will be independent of i , which finally results in optimum time-variant step-size. However, this assumption

used by Slock may not hold. We use identical (time-variant) step-size for individual

tap coefficient error variance to verify it. Based on NLMS algorithm, Eq.(3.11) can be

rewritten as

2 2 every tap converges slower than using individual step-size i.e., Slock’s algorithm has slower convergence rate than the ONLMS algorithm.

The optimum time-& tap-variant step-size can be obtained by taking derivative of Eq.(3.11) with respect to ( )u n and setting the result equal to zero. _i

Thus we can get the optimum time- & tap-variant step-size

, 1

The step-size obtained from this approach is optimum in the sense that, if it is used from n= to n k0 = , the tap coefficients error at the time instant n k= is minimum.

Substitute Eq.(3.14) into Eq.(3.11), we can get a recursive formula for the tap coefficient error variance g n as follows: _i( )

2

The ONLMS algorithm is summarized by the following equations:

1. e n( )=d n( )−w n x nˆ^T( ) ( )

We will use ONLMS algorithm in the following discussion.

3.4 Practical ONLMS algorithm

We have derived Optimum time- & tap-variant step-size for NLMS algorithm in Eq.(3.15). But it is useless since it requires prior knowledge of RIR w due to _i

w

: en velop h

Assume the RIR w can be modeled as an exponential decay envelope _i h _i

shown in Figure 3.1. Let decay envelop function be:

0 i

hi =hγ for i=1, ,L− (3.18) 1 where γ is the room exponential decay factor.

an exponential RIR model can be described as follows:

1. Measure room exponential decay factor γ to get h_i=h₀γⁱ 2. Set up initial value g_i(0)≈ for 0, , -1h_i² i= … L

3. ₂

By using the exponential RIR model, we can implement the ONLMS algorithm.

3.5 Simplified ONLMS algorithm

Using optimum time- & tap-variant step-size indeed increases convergence rate.

However, this algorithm has a higher computational cost than the conventional LMS algorithms.

In order to reduce the computational load, we propose another simplified version of ONLMS algorithm which uses piecewise time-variant and tap-variant step-size.

The method of choosing the step-size is described as follows:

1. 0 n n≤ ≤ (initial state): At ₀ n= , from Eq.(3.15) we obtain 0

Eq.(3.21) satisfied Eq.(3.15) only at n= . Convergence rate is fast only at the 0 Convergence rate is fast only at steady state.

3. For transient state n⁰≤ ≤ , we set n n¹ u proportional to _i g^{1/ 2}_i (0) i.e.,

Figure 3.2 Piecewise time-variant and tap-variant step-size for different state

In conclusion, Figure 3.2 illustrates three appropriate choices of step-size in different state. This results in piecewise time-variant and tap-variant step-size which reduces the computational load. However, simplified ONLMS does not perform as well as ONLMS.

3.6 Comparison of convergence

In this Section, we are trying to find out the reason that ONLMS has fastest convergence rate than other algorithms and why PNLMS and ESNLMS have two distinct slopes of convergence curve from the point of view of the tap coefficient error variance The average power of error signal ( )e n can be treated as convergence gain energy which is composed of all tap coefficient error variance, and can be expressed as [1]: step-size for large taps and small step-size for small taps from Eq.(2.8) and Eq.(2.9), respectively. This means large taps have large g n for these two algorithms at _i( ) initial state.

At the stage of initial adaptation, the power of error signal is dominated by the contribution from large taps (large g n ). Convergence rate is very fast because _i( ) large taps use large step-size. Eventually, however, the amount g n at these taps _i( ) reaches a point where it is becoming comparable to the amount g n from the _i( )

others small taps, the amount g n from small taps starts to dominate the power of _i( ) error signal. The convergence rate then slows down because small taps use small step-size. This phenomenon makes both ESNLMS and PNLMS algorithms have two distinct slopes of convergence curve. For PNLMS algorithm, initial rapidly period through experiment [4] will last until ERLE achieved ERLE_BREAK = −10log ( ) ₁₀ ρ where is defined in Eq.(2.9). Once ERLE_BREAK has been achieves, PNLMS is no longer offering any advantage over NLMS with a constant step-size.

Instead of just using large step-size, the ONLMS algorithm sets step-size directly proportionate to g n which is similar to PNLMS and ESNLMS algorithms at _i( ) initial state. Large g n uses large step-size to increase the convergence rate. After _i( ) the initial period, g n becomes identical for every tap, which means the ( )_i( ) J n is composed of every g n uniformly. From Eq.(3.15), identical ( )_i( ) g n results in ⁱ identical ( )u n thus convergence rate will not slow down. _i

In conclusion, ( )u n should be set proportional to _i g n for fast convergence _i( ) rate. Both PNLMS and ESNLMS algorithms use large step-size for large taps. This is appropriate since large taps indeed have large g n at initial stage. However, when _i( ) the values of g n for large taps are less than those for small taps, the convergence _i( ) rate then slows down. On the other hand, g n for every tap will become identical _i( ) for ONLMS algorithm after initial period. This results in identical ( )u n and will not _i slow down the convergence rate.

3.7 Computational complexity

We have already discussed several approaches of adjusting step-sizes which increase the convergence rate. However, using time variant or tap-variant step-size has high computational complexity compared with the constant step-size.

The usual way to measure algorithm complexity is to compute required numbers of multiplications. Under this measure, LMS algorithm has a complexity of order 2L since every sample every one of L impulse response estimated ˆ ( )w n must be _i multiplied by a delayed far-end signal to perform the convolution equation and these same delayed far-end signal samples must also be multiplied by a scaling of ( )e n to perform the correlation needed in the update equation. NLMS algorithm has a complexity of order 3L because in addition to the multiplies required before, squares of each of the (x n i− are needed to form ) x n . ( ) ²

In practice, NLMS is usually implemented as an order 2L algorithm with the simple trick of maintaining a running x n( ) ² by subtracting off the square of the outgoing sample. We assume this trick to be used by other algorithms as NLMS and the computational complexities of various algorithms are shown in Table 3.1.

We can find that the cost of increased convergence rate is the increased computation load. Traditional step-size adjustment approaches need L multiplications more compared to NLMS at every adaptation. ONLMS with has the fastest convergence rate needs additional 2L multiplications.

Algorithm Multiplications/sample

LMS 2L

NLMS 2L

VSNLMS 3L

ESNLMS 3L

PNLMS 3L

ONLMS 4L

Simplified

ONLMS 3L

Table 3.1 Computational requirements for different algorithms

Chapter 4