### Adaptive Decision Feedback Space–Time

### Equalization With Generalized

### Sidelobe Cancellation

*Yinman Lee, Member, IEEE, and Wen-Rong Wu, Member, IEEE*

**Abstract—In wireless communications, cochannel interference****(CCI) and intersymbol interference (ISI) are two main factors that**
**limit system performance. Conventionally, a beamformer is used**
**to reduce CCI, whereas an equalizer is used to compensate for**
**ISI. These two devices can be combined into one as space–time**
**equalizer (STE). A training sequence is usually required to train**
**the STE prior to its use. In some applications, however, spatial**
**information corresponding to a desired user is available, but the**
**training sequence is not. In this paper, we propose an adaptive**
**decision feedback STE to cope with this problem. Our scheme**
**consists of an adaptive decision feedback generalized sidelobe **
**can-celler (DFGSC), a blind decision feedback equalizer (DFE), and a**
**channel estimator. Due to the feedback operation, the proposed**
**DFGSC is not only superior to the conventional generalized **
**side-lobe canceller but also robust to multipath channel propagation**
**and spatial signature error. Theoretical results are derived for **
**op-timum solutions, convergence behavior, and robustness properties.**
**With the special channel-aided architecture, the proposed blind**
**DFE can reduce the error propagation effect and be more stable**
**than the conventional blind DFE. Simulation results show that the**
**proposed STE is effective in mitigating both CCI and ISI, even in**
**severe channel environments.**

**Index Terms—Blind equalization, channel estimation, decision****feedback equalizer (DFE), generalized sidelobe canceller (GSC),**
**space–time equalizer (STE).**

I. INTRODUCTION

**I**

N WIRELESS communications, the cochannel interference
(CCI) due to multiple access and the intersymbol
interfer-ence (ISI) due to multipath channels often cause severe signal
distortion and limit system performance [1], [2]. In recent
years, there has been a growing interest in applying adaptive
antenna arrays and space–time signal processing techniques
to solve these problems [3]–[5]. The common approach is to
use a beamformer for CCI reduction and an equalizer for ISI
compensation. These two devices can further be combined into
one as space–time equalizer (STE) [6], [7]. The application of
Manuscript received February 5, 2007; revised July 26, 2007, October 8, 2007, and November 12, 2007. This work was supported by the National Science Council, Taiwan, R.O.C., under Grant NSC 96-2219-E-009-018. The review of this paper was coordinated by Prof. J. Choi.

Y. Lee is with the Graduate Institute of Communication Engineering, National Chi Nan University, Nantou 545, Taiwan, R.O.C. (e-mail: ymlee@ ncnu.edu.tw).

W.-R. Wu is with the Department of Communication Engineering, National Chiao Tung University, Hsinchu 300, Taiwan, R.O.C. (e-mail: wrwu@faculty.nctu.edu.tw).

Digital Object Identifier 10.1109/TVT.2007.914472

the STE is beneficial to communication quality, subsequently improving the detection performance, even in severe channel environments.

The optimum STE is known to be a maximum likeli-hood sequence estimator (MLSE) operated in the space–time domain [7]. However, the MLSE is notorious for high com-putational complexity. A suboptimum approach with a hybrid of a linear filter and a Viterbi equalizer was proposed in [8]. Even so, the implementation complexity is still high, and it limits the MLSE-like structure in real-world applications. The STE performing both beamforming and equalization, as aforementioned, is what researchers consider most. The general form of the STE consists of an antenna array and a temporal filter bank [9], [10]. Either a linear equalizer or a decision feedback equalizer (DFE) can be applied to the structure. Although the performance of this structure is satisfactory, its computational complexity is quite high. The other problem is that when an adaptive algorithm is applied, the convergence is slow, particularly operating under a large number of antenna elements and a severe fading channel. To ease these problems, another form that is a hybrid (or cascade) of a spatial filter and a temporal filter was proposed for the STE [11], [12]. It requires lower computational complexity, and the convergence is faster. However, the space–time information of the received signal will not be fully exploited, and there will be some perfor-mance loss.

All the aforementioned STEs are training based. In other
words, we have to transmit training sequences before actually
using them. It is well known that the transmission of training
sequences will reduce the bandwidth utilization efficiency and
may not always be possible. In some applications, however,
spatial information corresponding to a desired user is available.
For instance, a receiver may perform direction of arrival (DOA)
estimation before signal detection. Moreover, a space division
multiple access (SDMA) system employed to increase system
capacity [13]–[16] transmits or receives a signal only from a
certain direction. A practical example is the broadband wireless
access system, particularly for fixed wireless applications. In
*these cases, it is possible to utilize the a priori spatial *
infor-mation and avoid the requirement of training sequences. A
straightforward idea is to use a generalized sidelobe canceller
(GSC) [17] for CCI suppression and a blind equalizer for ISI
compensation. Unfortunately, the result of this direct cascade
is often far from satisfactory. The reasons are stated in the
subsequent sections.

For computational complexity consideration, the GSC is often implemented with an adaptive structure. The least-mean-square (LMS) algorithm is a well-known and widely used adaptive algorithm [18]. However, the LMS-based adaptive GSC usually converges slowly. The error signal for the LMS adaptation in this case consists of the desired signal component, and so, it is large, even in a noiseless environment. This large error signal significantly magnifies the mean-squared error (MSE). To reduce the MSE, the step size (a parameter con-trolling the LMS convergence) must be small. This essentially makes the LMS algorithm converge slowly. In addition, the adaptive GSC is sensitive to constraint mismatch, which is caused by incorrect spatial information. In this case, the signal cancellation phenomenon will occur, and the performance of the adaptive GSC can seriously be degraded. In typical appli-cations, constraint mismatch can easily arise due to multipath channels and spatial signature errors.

It is well known that the DFE can have much better
perfor-mance than the linear equalizer in severe fading channels. This
statement is also true for blind equalization. However, the blind
DFE is difficult to derive and rarely reported in the literature.
*The major contribution in this field is from Labat et al., who*
proposed an interesting blind DFE in [19]. They used an infinite
impulse response (IIR) whitening filter cascaded with a blind
finite impulse response (FIR) linear equalizer in the startup
period. After convergence, it switches the cascading order of
the IIR and FIR filters, yielding a decision feedback structure.
At the same time, a decision-directed minimum MSE (MMSE)
training is initiated. For easy reference, we call this blind DFE
the LBDFE hereafter. One inherent problem associated with
the DFE is its error propagation effect, and this will have
even more impact in its adaptive implementation. Since the
LBDFE uses decision-directed training, it is sensitive to error
propagation.

In this paper, we propose an adaptive STE for systems with
*a priori spatial information. The proposed structure comprises*
a decision feedback GSC (DFGSC), a blind DFE, and a channel
estimator. The DFGSC structure can eliminate the desired
sig-nal component from the error sigsig-nal in the LMS adaptation. As
a consequence, it not only improves CCI suppression but also
allows the simple point distortionless constraint to be robust
to multipath channel environments and spatial signature errors.
The proposed blind DFE adapts a channel-aided structure,
yielding better ISI compensation and higher resistance to error
propagation. We will demonstrate that the proposed blind DFE
performs better than the LBDFE. This paper is an extension
of [20], in which only additive white Gaussian noise (AWGN)
channel was considered.

This paper is organized as follows. In Section II, the space–time signal model for the STE is developed. The effect of both CCI and ISI on the desired signal is also explained. In Section III, we describe a straightforward approach of the con-ventional GSC and LBDFE in a hybrid manner. In Section IV, we propose the new adaptive STE and describe the corre-sponding operation mechanisms in detail. Section V shows that the proposed STE is robust to general space–time multipath channels and spatial signature errors. Finally, simulation results and conclusions are given in Sections VI and VII, respectively.

II. SPACE–TIMESIGNALMODEL

*Consider a uniform linear array (ULA) of N antenna *
*el-ements at the receiver. Let the N* *× 1 received equivalent*
complex baseband signal vector in continuous time be
den-oted by

* x(t) = [x*0(t)

*x*1(t)

*· · · xN−1(t)]T*(1)

*where xn(t) (0≤ n ≤ N − 1, for all n hereafter) is the signal*

*received from the nth antenna element at time t, and the *
super-script (*·)T* _{denotes the transpose operation. Define an N}_{× 1}

**vector a(θ) as the spatial signature for the signal from the DOA****θ. It is written as a(θ) = [1 e**iζθ_{e}i2ζθ· · · ei(N−1)ζθ_{]}*T*_{, where}
*i =√−1, and ζθ= (2πd/Lλ) sin θ, in which d is the element*

*spacing, and Lλ* is the signal wavelength. Assume that there

*are M sources (including both desired and interfering sources)*
*coming from M different and distinguishable directions. The*
*transmitted signal waveform of the mth source sm(t) (0≤*

*m≤ M − 1, for all m hereafter) can be written as*
*sm(t) =*

+*∞*

*k=−∞*

*p(t− kT )bm(k)* (2)

*where bm(k) is the kth complex information symbol of the mth*

*source, p(t) is the pulse shape of the transmitted symbol, and T*
*is the symbol duration. Let Lm*be the number of propagation

*paths for the mth source, θmbe the DOA for the mth source,*

*τmlbe the delay time for the lth path (0≤ l ≤ Lm− 1, for all*

*l hereafter) of the mth source, and ¯hml*be the complex channel

*coefficient for the lth path of the mth source, respectively.*
Assume that the channel parameters for different sources are
independent and remain constant over the observation period.
The received signal vector in (1) can then be expressed as

**x(t) =***M−1*_{}
*m=0*
*L**m−1*
*l=0*
¯
*hmlsm(t− τml )a(θm) + n(t)* (3)

* where n(t) is an N× 1 complex AWGN vector. Each *
compo-nent in the AWGN vector is spatially white with a variance of

*σ*2

*n*. Substituting (2) into (3), we have

**x(t) =***M*_{}*−1*
*m=0*
+*∞*
*k=−∞*
**h***m(t− kT )bm (k) + n(t)* (4)

**where h***m(t) is an N× 1 vector summarizing the total *

*trans-mission effect of the mth source on the information, and it is*
given by
**h***m(t) =*
*L**m−1*
*l=0*
¯
*hmlp(t− τml )a(θm).* (5)

*Without loss of generality, we assume that the first M0*sources
*(M0≤ M) are related to the desired user, i.e., b*0(k) = b1(k) =
*· · · = bM*0*−1(k). The DOA θ0*is the DOA of the desired signal
*with the strongest signal strength, and θ1to θM*0*−1*correspond
to the DOAs of the desired signal’s spatial ISI. The received
equivalent complex baseband signal is sampled at the symbol
*rate, i.e., t = kT . The sampling clock is synchronized with the*
transmission clock. After sampling, the channel effect for all

Fig. 1. Hybrid of GSC and LBDFE (with LBDFE in startup period).

*sources is of a finite duration within [0, (Dm− 1)T ], where*

*Dm* *is the channel order of the mth source. Here, we define*

**d(k) as an N**× 1 vector representing the components from

**the main source of the desired signal (m = 0), and i(k) as***an N× 1 vector summing up the components from the spatial*
*ISI sources (m = 1, 2, . . . , M*0*− 1), and they are, respectively,*
given by
**d(k) =***D*0*−1*
*d=0*
**h**0*(d)b*0*(k− d)* (6)
**i(k) =***M*0*−1*
*m=1*
*D**m−1*
*d=0*
**h***m(d)b*0*(k− d).* (7)
*In addition, define an N × 1 vector z(k) representing the*
uncorrelated CCI-plus-noise components as

**z(k) =***M*_{}*−1*
*m=M*0
*D**m−1*
*d=0*
**h***m(d)bm(k − d) + n(k).* (8)

Then, the expression in (4) can be rewritten as

* x(k) = d(k) + i(k) + z(k).* (9)
We use this signal model to describe various kinds of CCI
and ISI in the wireless communication environment. The main
task of the STE is to suppress interference and recover the
transmitted information. To simplify the notations, we write

*b(k) instead of b*0(k) for the desired user’s information symbols in the following derivation.

III. HYBRID OFGSCANDLBDFE

In some wireless communication systems, such as SDMA
applications, the main DOA (and hence the spatial signature)
*of the desired signal is known a priori or can be estimated.*
A straightforward approach, as mentioned before for CCI and
ISI mitigation, is the hybrid of a conventional GSC and an
LBDFE. The purpose of the conventional GSC is to suppress
CCI, whereas that of the LBDFE is to compensate for ISI. With
the spatial signature, no extra training sequence is required for
either processing. The operation and weakness of this approach
will subsequently be elaborated upon.

*A. GSC*

The conventional GSC for CCI suppression is optimized with the linearly constrained minimum variance (LCMV)

*criterion [21]. The LCMV beamformer determines the N -tap*
**weight vector w through**

min

**w** **w**

*H*_{R}

**xw** subject to **C***H***w = f** (10)
**where Rx**= E**{x(k)x**H_{(k)}_{} is the input correlation matrix,}

**C is an N****× U constraint matrix, and f is a U × 1 response**

*vector, with U being the number of constraints. The superscript*
(*·)H* denotes Hermitian transposition. With the structure of
GSC, as shown in the left part of Fig. 1, the constrained
optimization problem can be transformed into an unconstrained
optimization problem [17]. This structure can effectively reduce
the computational cost, particularly when implemented with
adaptive algorithms. As illustrated in the figure, the upper path
* includes an N -tap quiescent signal matched filter wq*. The

*lower path includes an N × (N − U) blocking matrix B and*

*an (N*. Then, we have

**− U)-tap interference canceling filter w**a**an equivalent spatial filter as w = w***q − Bwa*. Ideally, the span

**of B is in the null space of C***H*_{. Using the constraint in (10),}

**w***q* **can readily be found to be w***q* **= C(C***H***C)***−1***f , and w***a* is

optimized according to the output power of the GSC as
*J = E*
*|ys(k)|*2
= E**(w***q − Bwa*)

*H*2

**x(k)***.*(11) The constrained optimization problem in (10) can then be rewritten as the following unconstrained optimization problem:

min
**w***a*

*J = min*
**w***a*

**(w***q − Bwa*)

*H*

**Rx(w**

*q*(12)

**− Bw**a).**The optimum w***a*is classically solved to be [17]

**w***a,opt* **= (B***H***RxB)***−1***B***H***Rxw***q.* (13)

*With the optimum weight vector, the minimum value of J in*
*(11), which is denoted as Jmin* (which equals the minimum
*output power, which is denoted as Po,min* for the conventional

GSC), can be calculated as

*J*min*= Po,min* (14)

**= w***H*_{opt}**Rxw**opt (15)
**= w***H _{q}*

**Rxw**opt (16)

**where we let wopt= w**

*q*

**− Bw**a,opt**. Since C and f are design**

**parameters, w***q***and B can be calculated offline. The calculation**

**of w***a,opt* shown in (13), however, is much more involved. An

alternative to find it is to use the adaptive training method. The LMS algorithm, being one of the stochastic gradient methods,

is chosen here for its simple yet effective nature. Taking the
* sto-chastic gradient of J with respect to w∗_{a}*, where (

*·)∗*denotes the conjugate operation, we can obtain the LMS update equation

**for w**

*a*as

**w***a (k + 1) = wa(k) + μav(k)e∗s(k)* (17)

**where w***a (k) is the estimate of wa,opt*

*at the kth iteration,*

**v(k) = B**H_{x(k) is the filter input vector (the output vector}

*from the blocking matrix), μa* is the step size controlling the

*convergence rate, and es(k) is an error signal between the *

*de-sired and actual output. For GSC applications, we have es(k) =*

*ys (k). When wais optimized, the error signal es(k) will chiefly*

include the component from the desired signal, i.e., the desired
*user’s transmitted information symbols b(k). This indicates that*
* the stochastic gradient in (17), i.e., v(k)y∗_{s}(k), will be large,*
even for optimum weights. When the LMS algorithm is applied

**to estimate w**

*a,opt*, the performance will be affected due to the

large error signal used. The other problem with the conventional
GSC is its sensitivity to constraint mismatch. Whenever the
**setting of the constraint matrix C in (10) [or the blocking matrix**

**B in (12)] is not fit for the actual spatial signature of the desired**

signal, constraint mismatch occurs. Constraint mismatch can
easily arise due to multipath channels and spatial signature
errors. This seriously degrades the performance of the adaptive
GSC. All these problems will be analyzed in depth later.
*B. LBDFE*

The output of the GSC is fed into the LBDFE for
equal-ization. The equalizer adaptation process is divided into two
periods. In the startup period, the received signal is prewhitened
by an IIR filter and then equalized by a blind FIR linear
equalizer. The structure of the LBDFE in the startup period
is shown in the right part of Fig. 1. In [19], the constant
modulus algorithm (CMA) was used as the blind algorithm for
equalization. Recently, a sophisticated blind equalization
algo-rithm called the multimodulus algoalgo-rithm (MMA) was proposed
[22]. Analysis shows that the MMA can provide much more
stable performance, particularly with high-order constellation
modulation [23]. The cost function can yield an equalized
constellation rotated with a multiple of 90*◦*, eliminating the
need for additional constellation phase recovery as needed in
the CMA. The remaining phase ambiguity problem is
classi-cally solved by differential encoding. For these reasons, we
use the MMA as our blind equalization algorithm (instead of
the CMA) throughout this paper. For the LBDFE, we denote
**the length of w***f* **and w***b* *as α and β, respectively. In the*

*startup period, let ys(k) be the output of the GSC, i.e., ys(k) =*

**(w***q −Bwa*)

*H*

**x(k), as given in (11), and let y**s(k) be the*difference between ys(k) and the prewhitening filter output.*

**In addition, let y****(k) be the input vector of w**b**as y***(k) =*

*[y _{s}(k−1) y_{s}(k−2) · · · y_{s}(k−β)]T. So, we have y_{s}(k) =*

*ys(k)*

**− w**Hb**y**

*is optimized*

**(k). The filter w**baccord-ing to the criterion minw*b* E*{|ys(k)|*

2_{}. As shown in}*Fig. 1, ys(k) serves as the input of the blind FIR *

**fil-ter w***f* **as well. We define another vector as y***(k) =*

*[y _{s}(k)*

*y*

_{s}(k−1) · · · y_{s}(k−α+1)]T_{, and, as a }

conse-quence, the output of the LBDFE in the startup period

*is yt (k) = wHf*

**y**

*(k). For the MMA, the error signal is*

defined as
*et(k) =*
*y*2* _{t,r}(k)− R*22+

*y*2

*22 (18) with*

_{t,i}(k)− R*R*2= E

*b*4

*E*

_{r}(k)*{b*2

*r(k)}*= E

*b*4

*E*

_{i}(k)*{b*2

*i(k)}*(19)

*in which the subscripts r and i in yt(k) and b(k) denote the real*

*and imaginary parts of yt(k) and b(k), respectively. The cost*

**function for the optimization of w***f*is then written as

*J*MMA= E*{et(k)}*

= E*y*2* _{t,r}(k)−R*22+

*y*2

*22*

_{t,i}(k)−R*.* (20)
*Let μf***be the step size controlling the convergence of w***f*. From

**[22], the stochastic gradient algorithm for the MMA of w***f*can

be obtained as

**w***f (k + 1) = wf(k) + μfφ (yt(k)) y(k)* (21)

with

*φ (yt(k)) = yt,r*3 *(k) + iy*3*t,i(k)− R*2*yt(k).* (22)

The startup period is expected to sufficiently open the eye pattern such that the error rate is low enough to initiate the second (tracking) period. In the tracking period, the cascading order of the IIR prewhitening filter and the FIR linear equalizer is swapped, and this turns the whole system into a conventional DFE structure. A decision-directed MMSE tracking operation, similar to that of the conventional DFE, is then activated. This approach may initially avoid the possible error propagation phenomenon and give a smooth transition strategy between blind and decision-directed equalization. Nevertheless, the sta-bility of the adaptive IIR filter makes the performance sensitive to the parameters chosen. In addition, error propagation may occur during the decision-directed mode. The behavior of the LBDFE becomes not easy to control. We will empirically show in Section VI that the performance of the hybrid of the conventional GSC and LBDFE is often far from satisfactory in severe channel environments.

IV. PROPOSEDHYBRIDSTE

In this section, we propose a new adaptive STE that is a
hybrid of DFGSC and a channel-aided blind DFE (CBDFE).
Fig. 2 shows the block diagram of the proposed STE. Note that
in Fig. 2, a channel estimator is included. Let the coefficients
**of the channel estimator be denoted as w***h*. It is used to

model the equivalent temporal channel for the desired signal, and its operation will be explained soon. In [11], a channel estimator was also used in a training-based STE such that the corresponding beamformer can achieve better performance. In the STE scheme proposed here, the role of the channel estimator is much more involved. For spatial processing, with the help of the channel estimator, we can formulate the adaptive DFGSC,

Fig. 2. Proposed hybrid STE.

achieving better CCI suppression performance. In addition, the DFGSC can have extra robustness against constraint mismatch. For temporal processing, with the help of the channel estimator, we can formulate the CBDFE such that it is more effective in ISI compensation. This CBDFE is different from those channel-estimation-based DFEs proposed in [24] and [25], where the feedforward and feedback filters are calculated based on the estimated channel response. In the CBDFE, however, the adap-tive structure is preserved. The operation of the DFGSC and CBDFE is separately presented as follows.

*A. Proposed DFGSC*

**Let the channel estimator w***h* *have a dimension of γ× 1.*

*Here, the value of γ is chosen to be equal to or larger than the*
*maximum value of the channel order Dm*(0*≤ m ≤ M*0*− 1).*
We first define ˆ* b(k) as the input vector of wh*, i.e., ˆ

**b(k) =**[ˆ*b(k)* ˆ*b(k− 1) · · · ˆb(k − γ + 1)]T*_{. To optimize the }

**inter-ference canceling filter w***a* **and the channel estimator w***h*, we

propose a new cost function as
*J = E*
*|es(k)|*2
= E
*ys(k) − wHhb(k)*ˆ
2
= E

**w**

*Hq*

**x(k)**−**w**

*Ha*

**w**

*Hh*

**B**

*H*ˆ

_{x(k)}*2*

**b(k)***.*(23)

To understand the operation mechanisms, we first study a
simplified scenario in which only the temporal ISI is present.
In other words, the desired signal and its ISI only come from
*the main DOA, i.e., M0 = 1 (and so i(k) in (7) is a zero*
vector). Consider that the spatial signature for the desired signal
is exactly known, and a distortionless constraint is set toward

*it. Thus, U = 1 is used in the derivation. In this case, the*desired signal will be completely blocked in the lower path of the DFGSC. The general case where spatial ISI exists and the distortionless constraint may be set improperly will be discussed in the next section. Assuming that the decision is correct, i.e., ˆ

*b(k) = b(k), we can rewrite the minimization of*the cost function in (23) as

min
**w***a ,wh*

*J = min*

**wc**

**w**

*H*

_{q}**Rxw**

*q*

**− w**Hq**[ RxB**

**Rp]wc**

**−w**H**c**

**B**

*H*

**Rx**

**R**

*H*

**p**

**w**

*q*

**+ w**

*H*

**c**

**Rcwc**(24) where we let

**wc**=

**w**

*a*

**w**

*h*(25)

**Rc**= E

**B**

*H*ˆ

**x(k)**

**b(k)****[ x**

*H*

_{(k)B}**ˆ**

_{b}*H*=

_{(k) ]}**B**

*H*

_{R}**xB**

**0**

**0**

*H*

*2*

_{σ}*b*

**I**

*γ*(26)

**Rp**= E

**x(k)ˆ****b**

*H(k)*

*= σ*2

_{b}**[h0(0)**

**h**0(1)

*0(γ*

**· · · h***− 1)]*(27)

**in which 0 denotes a zero matrix with dimension (N**− 1) × γ,**I***γ* *denotes an identity matrix with dimension γ× γ, and σ*2*b*

denotes the power of transmitted symbols. Since there is no
**correlation between B***H_{x(k) and ˆ}_{b(k) in this case, the }*

**off-diagonal block of Rc** **in (26) is zero. Again, w***q* is for signal

matching, and it can be set the same as for the conventional
GSC. Taking the derivative of this cost function with respect
**to w***∗*** _{c}** and setting the result to zero, we can obtain the

**opti-mum wc**as

*∂J*

**∂w**∗**=**

_{c}*−2*

**B**

*H*

**Rx**

**R**

*H*

**p**

**w**

*q*

**+ 2Rcwc**

*= 0.*(28) Thus

**w**

**c,opt****= R**

*−1*

**c**

**B**

*H*

_{R}**x**

**R**

*H*

**p**

**w**

*q.*(29)

**With the special structure of Rc, we can decompose w*** c,opt*
back into the two weights

**w***a,opt***= (B***H***RxB)***−1***B***H***Rxw***q* (30)
**w***h,opt*=
**R***H*
**p**
*σ*2
*b*
**w***q*
**= [h**0(0) **h**0(1)* · · · h*0

*(γ− 1)]H*

**w**

*q.*(31)

**From (30), we observe that the expression of w***a,opt*is the same

**as that of the conventional GSC given in (13). Since w***q* and

**w***a,optremain the same, the minimum output power Po,min*for

**the DFGSC is also the same. From (31), we observe that w***h,opt*

**the DFGSC output. With w***a,opt* **and w***h,opt* given earlier, the

*minimum J in (23) for the DFGSC becomes*
*J*min= E
**w***H*
opt**x(k)****−w**h,optH* b(k)*ˆ
2

**= w**

*H*

_{q}**Rxw**opt

*−σb*2

**w**

*Hq*

**[h0(0) h0(1)**

*0(γ*

**· · · h***−1)]*2

*= Po,min−σ*2

*b*

**w**

*Hq*

**[h**0

**(0) h**0(1)

*0*

**· · · h***(γ−1)]*2

*.*(32)

*Note here that J*min

*is no longer equal to Po,min*

**. We let Rd**= E

**{d(k)d**H_{(k)}_{} be the input correlation matrix of the desired}**signal excluding spatial ISI and Rz**= E* {z(k)zH(k)} be the*
input correlation matrix of CCI-plus-noise. The first term in
(32) is equal to the total power in the DFGSC output, and the
second term in (32) is the power of the desired signal in the
DFGSC output, which can be defined as

*P***d**
Δ
**= w***H*_{opt}**Rdw**opt
*= σ _{b}*2

**w**

_{q}H**[h**0(0)

**h**0(1)

*0*

**· · · h***(γ− 1)]*2

*.*(33)

*So, Jmin*in (32) can be calculated as

*J*min*= Po,min− P***d** (34)
**= w***H _{q}*

**Rzw**opt

*.*(35) Comparing (14) and (34), we can see that the desired signal is

**totally excluded with the help of w**

*h,opt*. Thus, we conclude that

*the decision feedback only reduces the minimum J in (14), and*
the optimum performance is not enhanced. When an adaptive
**algorithm such as LMS is used to estimate w***a,opt*, however,

the performance of the GSC can greatly be improved by the feedback operation. The LMS update equations for the DFGSC can be written as

**w***a (k + 1) = wa(k) + μav(k)e∗s(k)* (36)

**w***h (k + 1) = wh(k) + μhb(k)e*ˆ

*∗s(k)*(37)

*where μa* **is the step size for w***a, μh* is the step size for

**w***h , v(k) is the output from the blocking matrix, and es(k) =*

*ys(k) − wHh(k)ˆb(k). Unlike the conventional GSC, the *

*steady-state es(k) will exclude the desired signal, and hence, it can*

be quite small. This is where the improvement of the adaptive DFGSC stems from.

To analyze the steady-state performance for both the
con-ventional adaptive GSC and the proposed adaptive DFGSC, we
*denote the value of J [both (11) and (23)] in the steady state as*
*J (∞). Suppose that the decision is correct, and wh*is fixed at

optimum. Then

*J (∞) = J*min*+ Jex(∞)* (38)
*where J*ex(*∞) is the excess MSE. Note that the excess MSE is*
yielded by the use of the LMS algorithm. In addition, define the
weight error vector as

* (k) = wa(k)− wa,opt.* (39)

Using the direct averaging method [18], we can have

**(k + 1) = (I**− μa**B***H***Rx****B)(k)+ μ**a**B***H x(k)e∗*opt

*(k)*(40)

*where e*opt

*(k) denotes the error signal produced with the*opti-mum weights. Define the correlation matrix of the weight error vector as

**K(k) = E****(k)**H(k)*.* (41)
Invoking the independence assumption [18], we can obtain the
**recursive relation of K(k) as**

**K(k + 1) = (I**− μa**B***H***Rx****B)K(k)(I**− μa**B***H***RxB)**
*+ μ*2*aJ*min**B***H***Rx*** B.* (42)
Under this premise, the excess MSE, which is denoted as

*J*ex(k), is written as

*J*ex(k) = tr
**B***H***Rx****BK(k)**

(43)
where tr[*·] gives the trace of the matrix inside the brackets. As*
*k→ ∞, J*ex(k) is given by
*J*ex(*∞) = J*min
*N*_{}*−1*
*l=1*
*μaλl***(B***H***RxB)**
2*− μaλl***(B***H***RxB)**
(44)

*where λl***(B***H***Rx****B) represents the lth eigenvalue of B**H**RxB.**
*If μa*is small, (44) can be approximated as

*J*ex(*∞) ≈*
*μaJ*min
2
*N**−1*
*l=1*
*λl***(B***H***Rx*** B).* (45)

*Thus, Jex(∞) is proportional to J*min

*and the step size μa*. The

transient output signal-to-interference-plus-noise ratio (SINR) for the conventional GSC and DFGSC can be written as

*SINR(k) =*

E**w***H_{(k)d(k)}*2

E

* |wH_{(k)x(k)}_{− w}H_{(k)d(k)}_{|}*2

*.*(46)

The optimum and steady-state output SINR (with the LMS algorithm) can then be calculated as

SINRopt=
E**w***H*
opt* d(k)*
2
E

**w**

*H*opt

*opt*

**x(k)****− w**H*2 =*

**d(k)***P*

**d**

*Po,min− P*

**d**(47) SINRLMS= E

**w**

*H*(

*2 E*

**∞)d(∞)**

**|w**H_{(}

**∞)x(∞) − w**H_{(}

*2 =*

**∞)d(∞)|***P*

**d**

**w**

*H*

*q*

**Rxw**opt

**+ tr [B**

*H*

**RxBK(**

*∞)] − P*

**d**=

*P*

**d**

*Po,min+ J*ex(

*∞) − P*

**d**

*.*(48)

**Note that the notations w(*** ∞), d(∞), and x(∞) are used to*
denote their final values and are based on the assumption of

reaching convergence under statistical expectation, just like
*that for Jex(n)→ J*ex(*∞) as n → ∞. From (47), once again,*
we see that the optimum performance is not enhanced by the
*decision feedback operation since Pdand Po,min*are the same

for the conventional GSC and the DFGSC. From (45), we can
*see that Jex(∞) is proportional to J*min. The corresponding
*J*min values for both schemes are shown in (14) and (34),
*respectively. The resultant Jmin*for the adaptive DFGSC will
be smaller than that for the conventional adaptive GSC. As a
*consequence, Jex(∞) for the adaptive DFGSC will be smaller*
than that for the conventional adaptive GSC. Moreover, from
*(48), we see that the smaller J*ex(*∞) is, the larger the *
steady-state output SINR we will have. We thus conclude that with
the same step size, the steady-state output SINR of the adaptive
DFGSC will be higher than that of the conventional adaptive
GSC. In addition, note that the step size bound for the LMS
algorithm is determined by the eigenvalue spread of the input
correlation matrix. The input vectors for the conventional GSC
and the DFGSC are the same, and so, the step size bounds for
both schemes are the same.

*B. Proposed CBDFE*

The MMA is conventionally applied to the blind FIR linear equalizer only. Since the DFE is a nonlinear and IIR equalizer, direct application of the MMA may result in severe error propa-gation in the startup period. It may make the performance of the blind DFE even worse than that of its FIR linear companion. Here, we make use of the previously derived channel estimator

**w***h*and propose a new structure, i.e., the CBDFE, to overcome

this problem. In FIR linear equalization, the performance
of the MMA can be demonstrated to approach that of the
training-based MMSE method [19], [23]. Hence, we suppose
that the weights solved by the MMA are close to those solved
by the MMSE criterion. Our approach uses a basic principle of
the DFE, i.e., the postcursor response of the channel convolved
with the feedforward filter is cancelled by the feedback
filter.1 **Let w***f* **and w***b* be the feedforward and feedback filter

*weight vectors of a conventional DFE, with length α and β,*
*respectively. Similar to the previous section, ys(k) denotes the*

**output of the DFGSC. For notation simplicity, the input of w***f*

**and w***b* **is again written as y(k) = [y**s(k)*ys(k− 1) · · ·*

*ys(k− α + 1)]T* and ˆ**b(k) = [ˆ**b(k− κ − 1) ˆb(k − κ−2) · · ·

ˆ_{b(k}_{− κ − β)]}T_{, where κ is the decision delay. Note that}

the convolution of the equivalent channel response and the
*feedforward filter results in a response of length α + γ− 1.*
Thus, for perfect postcursor cancellation, we must have
*β≥ α + γ − 2 − κ. Without loss of generality, we let*
*β = α + γ− 2 − κ. We now prove the postcursor cancellation*
property mentioned earlier. Again, with correct decisions, the
error signal is written as

*et(k) = b(k− κ) − yt(k)*

*= b(k− κ) −***w***H _{f}y(k)− w_{b}Hb(k)* (49)

1_{For DFE, precursor and postcursor responses are defined as the ISI from}

future and past symbols, respectively.

where *yt(k)* is the equalizer output. Straightforward

manipulations give the equalizer output MSE as
E
*|et(k)|*2
**= w***fH***Ryyw***f − wfH*

**Rybw**

*b*

**− w**Hf**p**

**yb**

**− w**H*b*

**R**

*H*

**ybw**

*f*

**+ w**

*Hb*

**Rbbw**

*b*

**+ w**

*bH*

**p**

**bb**

**− p**H

**yb****w**

*f*

**+ p**

*H*

**bb****w**

*b+ σb*2 (50)

**with Ryy**= E**{y(k)y**H_{(k)}_{}, R}

**bb**= E**{b(k)b**H(k)**}, R****yb**=
E**{y(k)b**H_{(k)}_{}, p}

* yb*= E

*= E*

**{y(k)b**∗(k**− κ)}, and p****bb***MMSE criterion, we set the gradient of E*

**{b(k)b**∗(k− κ)}. To obtain the optimum weights with the*{|et(k)|*2

*} with*

**respect to the vectors w***∗ _{f}*

**and w**

*∗*to zero. This results in

_{b}**w***f,opt*=
**Ryy***−*
1
*σ*2
*b*
**RybR***H***yb**
*−1*
**p*** yb* (51)

**w**

*b,opt*= 1

*σ*2

*b*

**R**

*H*

_{yb}w*f,opt.*(52)

**Let w***h,opt= [ω0* *ω*1 *· · · ωγ−1*]*T* represent the convolution
**of w***h,opt* **and w***f,opt* **as Hw***f,opt***, where H is an**

*(α + γ− 1) × α matrix as*
**H =**
⎡
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
*ω*0 0 *· · ·* *· · ·* *· · ·* *· · ·* *· · ·* 0
*ω*1 *ω*0 0 *· · ·* *· · ·* *· · ·* *· · ·* 0
..
. . .. ...
*ωγ−1* *ωγ−2* *· · ·* *ω*0 0 *· · ·* *· · ·* 0
0 *ωγ−1* *ωγ−2* *· · ·* *ω*0 0 *· · ·* 0
..
. . .. . .. ...
0 *· · ·* 0 *ωγ−1* *ωγ−2* *· · ·* *ω*0 0
0 *· · ·* *· · ·* 0 *ωγ−1* *ωγ−2* *· · ·* *ω*0
..
. . .. ...
0 *· · ·* *· · ·* *· · ·* *· · ·* 0 *ω _{γ−1}*

*ω*0

_{γ−2}*· · ·*

*· · ·*

*· · ·*

*· · ·*

*· · ·*0

*ωγ−1*⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

*.*(53)

**We can further partition H as H = [H**

*T*

*r* **H***Tp*]*T***, where H***r*is

*of dimension (κ + 1) × α, and Hp*

*is of dimension (α + γ−*

2**− κ) × α. It is not difficult to see that H**r**w***f,opt*corresponds

**to the precursor response of Hw***f,opt***, whereas H***p***w***f,opt* is

the postcursor response. With enough degrees of freedom, CCI is mostly suppressed. The DFGSC output can be modeled as

*ys (k) = wHq*

*(54)*

**d(k) + ν(k)*** where ν(k) is a white noise independent of d(k). With (54)*
and some manipulations, we can derive

1
*σ*2

*b*

**R***H*_{yb}= H*p.* (55)

**From (52), we then obtain w***b,opt***= H***p***w***f,opt*. This result can

be restated as

**w***b,opt*= post* {wh,opt⊗ wf,opt}* (56)

where *⊗ denotes the convolution operation, and post{·}*
denotes the postcursor-taking operation. This result suggests

an adaptation approach for the training-based MMSE-DFE.
**Let w***f (k) and wb(k) be the feedforward and feedback weight*

*vectors at time instant k. With reference to (56), we can let*

**w***b(k) = post {wh(k)⊗ wf(k)}* (57)

**in which w***h(k) is the channel estimate at time instant k. If*

**w***h (k) converges to wh,opt*

**, w**

*b*

**(k) will converge to w**b,opttoo. The difference between this approach and the conventional
**method lies in that only w***f (k) is adapted [not both wf(k)*

**and w***b(k)]. While this approach may make no difference for a*

training-based DFE (since the training sequence is available), it
will provide significant improvement for the blind DFE. On one
**hand, consider a blind DFE scenario in which both w***f(k) and*

**w***b(k) are adapted. With yt (k) = wHf(k)y(k)− wHb*

*(k)ˆ*

**b(k),**the update equations for the MMA can be written as

**w***f (k + 1) = wf(k) + μfφ (yt(k)) y(k)* (58)

**w***b (k + 1) = wb(k)− μ*b

*φ (yt(k)) ˆ*(59)

**b(k)***where φ(yt(k)) is the same as that given in (22). From (58)*

and (59), we see that if there is a decision error, the error will
immediately reflect to ˆ**b(k) and then φ(y**t(k)). Note that the

**adaptation of w***f(k) only involves erroneous φ(yt(k)), whereas*

**that of w***b(k) involves both erroneous ˆ b(k) and erroneous*

*φ(yt (k)). The two error sources in (59) will make wb(k) quite*

sensitive to decision errors. On the other hand, in the proposed
**method, only w***f(k) is adapted, as given in (58). Although*

*the effect of decision error will also pass to es(k), which will*

**perturb the adaptation of w***a (k) and wh(k), the influence is*

**smaller. This is because the convergence of w***a (k) and wh(k)*

in the DFGSC is much faster and more stable than that of the
blind DFE (which will empirically be shown in Section VI). By
**using (57) to calculate w***b(k), the feedback filter of the blind*

DFE will perform much better. In one word, with the proposed operation, the resultant CBDFE will be less sensitive to decision errors.

In the startup period, decisions may not be trustworthy.
Including decisions in the operation of the CBDFE may affect
its stability. Unlike the LBDFE’s mode switching, we propose a
simple mechanism allowing a smooth transition from FIR linear
equalization to DFE. We let the feedback filter be multiplied
*by a time-varying factor f (k), where f (k)≤ 1. Initially, k is*
*small, and we set f (k) small such that the weighting of the*
feedback filter is small. In this stage, the proposed equalizer will
*behave much like an FIR linear equalizer. As k increases and*
*decisions become more reliable, we increase f (k). Eventually,*
*f (k) approaches one, and the equalizer becomes a full DFE.*
As known in [18], the convergence of an adaptive algorithm
exhibits exponential decay behavior. Thus, a natural choice for
*f (k) will be*

*f (k) = 1− e−ξk* (60)

*where ξ is a design parameter controlling the increase rate of*
this factor. With the proposed mechanism, the feedback filter is
gradually taken into account, and the error propagation effect

will be reduced. The CBDFE will ultimately approach the optimum MMSE-DFE.

V. SPATIALMULTIPATH ANDSPATIAL

SIGNATUREMISMATCH

In this section, we will show that the proposed STE is robust
to two unfavorable phenomena, i.e., spatial multipath
propa-gation and spatial signature mismatch, which are frequently
encountered in array signal processing. Let us first consider the
spatial multipath propagation problem. Multipath propagation
may cause ISI coming from different directions, which induces
both coherent interference [26] and angular spread [27]. In this
case, the conventional GSC with the simple point distortionless
constraint tends to cancel the desired signal itself, which is
referred to as signal cancellation. In the proposed STE, even
with this spatial ISI, the phenomenon of signal cancellation
will not exist. This is due to the use of decision feedback. In
the spatial multipath environment, the desired signal will leak
to the output of the blocking matrix, producing a correlation
between the upper and lower paths in the DFGSC. Thus, the
**optimum solutions for the interference canceling filter w***a*and

**the channel estimator w***h*are now coupled together. From (29),

we have
**w*** c,opt*=

**w**

*a,opt*

**w**

*h,opt*

**= R**

*−1*

_{c}**B**

*H*

_{R}**x**

**R**

*H*

**p**

**w**

*q*(61) where

**Rp**= E

**x(k)ˆ****b**

*H(k)*

*= σ*2

_{b}*0*

_{M}*−1*

*m=0*

**h**

*m*(0)

*M*0

*−1*

*m=0*

**h**

*m*(1)

*· · ·*

*M*0

*−1*

*m=0*

**h**

*m(γ−1)*(62)

**Rc**=

**B**

*H*

_{R}**xB**

**M**

**M**

*H*

*2*

_{σ}*b*

**I**

*γ*(63) with

*2*

**M = σ**_{b}**B**

*H*

*0*

_{M}*−1*

*m=0*

**h**

*m*(0)

*M*0

*−1*

*m=0*

**h**

*m*(1)

*· · ·*

*M*0

*−1*

*m=0*

**h**

*m(γ−1)*(64) which is the correlation matrix between the blocking matrix

**output B**

*H*

**x(k) and the decision vector ˆ****b(k). Using the**inver-sion identity for subblock matrices [28], we can find the inverse
**of Rc, and so, wc,opt**in (61), i.e.,

**w*** c,opt*=

**(B**

*H*

**RzB)**

*−1*

**B**

*H*

**Rzw**

*q*

*0*

_{M}*−1*

*m=0*

**h**

*m*(0)

*· · ·*

*M*0

*−1*

*m=0*

**h**

*m(γ− 1)*

*H*

*×*

**I**

**− B(B**H**RzB)**

*−1*

**B**

*H*

**Rz**

**w**

*q*(65)

**and thus the coupled w***a,opt***and w***h,opt*can be written as
**w***a,opt***=(B***H***RzB)***−1***B***H***Rzw***q* (66)
**w***h,opt*=
* _{M}*
0

*−1*

*m=0*

**h**

*m*(0)

*· · ·*

*M*0

*−1*

*m=0*

**h**

*m(γ− 1)*

*H*

*(67)*

**× (w**q**− Bw**a,opt).*The minimum J of the DFGSC, as that given in (35), can then*
be solved to be
*J*min**= w***qH***Rxw***q − wqH*

**[ RxB**

**Rp]R**

*−1*

**c**

**B**

*H*

_{R}**x**

**R**

*H*

**p**

**w**

*q*

**= w**

_{q}H**Rzw**

*q*

**− w**qH**RzB(B**

*H*

**RzB)**

*−1*

**B**

*H*

**Rzw**

*q*

**= w**

_{q}H**Rzw**opt

*.*(68)

**From (66)–(68), we see that while w***a,opt* **and w***h,opt* are

*changed, Jmin*(and so the error signal) still contains no desired
signal. The leaky desired signal in the lower path of the DFGSC
**is not correlated with the error signal. When optimizing w***a*, the

output desired signal power is not minimized, and so, no signal cancellation occurs.

We then consider the spatial signature mismatch problem. If the knowledge of the desired signal’s main DOA is erroneous,

**w***q* no longer matches the desired signal’s spatial signature,

**and B cannot obstruct the desired signal. In this case, signal**
cancellation may occur as well [29]. In the case of spatial
multipath propagation, we have already proven that for the
DFGSC, even if there is leaky desired signal in the output of
**the blocking matrix, w***a*will not cancel the desired signal. The

spatial signature mismatch can be considered a signal-leaking problem similar to that in spatial multipath propagation. Thus, no signal cancellation will occur in the DFGSC. The derivation details, however, are omitted.

For these scenarios, the expression of the optimum output
*SINR is the same as that given in (47), but now, P***d***and Po,min*

are changed. The output desired signal power becomes
*P***d***= σb*2
**(w***q − Bwa,opt*)

*H*

*×*

*0*

_{M}*−1*

*m=0*

**h**

*m*(0)

*M*0

*−1*

*m=0*

**h**

*m*(1)

*· · ·*

*M*0

*−1*

*m=0*

**h**

*m(γ−1)*2 (69)

*which is different from (33). In the calculation of Po,min*,

(15) should be used instead of (16). Moreover, due to the
existence of the desired signal in the input of the interference
canceling filter, the excess MSE yielded by the LMS algorithm
*will be different too. Let J*ex**d**(*∞) denote the excess MSE value*
for the desired signal in these leaky scenarios. Using (44), we
can have

*J*_{ex}**d**(*∞) = J*min

*μaλ*1(B*H***RdB)**
2*− μaλ*1(B*H***RdB)**

*.* (70)

The result in (70) is due to the fact that only one eigenvalue of

**B***H*_{R}

**dB is nonzero. The expression of the steady-state output**

TABLE I

PARAMETERSUSED INSIMULATIONS OFPARTA: (a) FILTERLENGTH AND(b) STEPSIZES

SINR for the GSC structure with spatial ISI or spatial signature mismatch is then slightly modified to

SINRLMS=

*P***d***+ J*ex**d**(*∞)*

*Po,min+ Jex(∞) − (P***d***+ J*ex**d**(*∞))*

(71)
*in which Pdand Po,min*should also be changed to use (69) and

(15), respectively.

VI. SIMULATIONS

Computer simulations are conducted to demonstrate the
ef-fectiveness of the proposed STE (called DFGSC-CBDFE in this
section for clarity) and verify our analytic results. For
com-parison, we also consider the hybrid of the conventional GSC
and LBDFE (called GSC-LBDFE hereafter) and the hybrid of
the DFGSC and LBDFE (called DFGSC-LBDFE hereafter).
Note that the latter scheme is used for the comparison of
CBDFE and LBDFE. In all cases, we assume a ULA with
five omnidirectional antennas spaced half a wavelength apart.
*The main DOA of the desired signal is known a priori. Only*
the point distortionless constraint to the main DOA is used
for the GSC in all three schemes. The received desired signal
is corrupted by CCI, ISI, and AWGN. In the first part, we
only consider channels with temporal ISI. In the second part,
we consider more realistic channels where both temporal and
spatial ISIs are present. In all figures, at least 500 simulation
runs are averaged to obtain each simulated result.

*A. Channels With Temporal ISI Only*

In this set of simulations, we consider channels with temporal
ISI only. We compare the performance of DFGSC-CBDFE with
GSC-LBDFE and DFGSC-LBDFE. The transmitted symbols
are randomly generated and then modulated by quadrature
phase-shift keying. The CCI-to-noise ratio (CCINR) is set as
25 dB, and the signal-to-noise ratio (SNR) is 15 dB. The
channel for the desired signal coming from 0*◦* is [0.407
0.815 0.407] [1, p. 616], and the channel for CCI
com-ing from *−60◦* is [1 0 0]. The parameters used for the
adaptive GSC, DFGSC, LBDFE, and CBDFE are listed in
**Table I. The step sizes for w***h* in those decision

feed-back schemes are 2*× 10−3*. The decision-directed MMSE
training starts after 500 iterations. As mentioned, the factor
*f (k) for the CBDFE is used to reduce error propagation.*
*In this case, ξ is chosen to be 0.01, and f (k) approaches*

Fig. 3. Learning curves of GSC output SINR for different schemes in sup-pressing CCI.

Fig. 4. Beam patterns (enlarged region around CCI’s DOA) of different schemes in Fig. 3 after 5000 iterations.

unity at around 500 iterations. Fig. 3 shows the GSC output
SINR for CBDFE, GSC-LBDFE, and
DFGSC-LBDFE. In addition, the optimum SINR and the steady-state
SINR obtained with the LMS algorithm are shown. From Fig. 3,
we see that all schemes are comparable in convergence rate,
but the SINR achieved by those schemes with the adaptive
DFGSC is higher than that with the conventional adaptive GSC.
As expected, the adaptive DFGSC can approach the optimum
solution much more closely, which means that the effect of
the excess MSE induced by the LMS algorithm is smaller. In
terms of GSC output SINR and convergence rate,
DFGSC-CBDFE provides the best performance. Fig. 4 reveals the beam
patterns (enlarging the region around the CCI’s DOA) of those
schemes used in Fig. 3 after 5000 iterations. The improvement
of the decision feedback operation can clearly be seen as well.
Fig. 5 shows the equalizer output MSE, i.e., E*{|b(k − κ) −*
*yt(k)|*2*}, for all three schemes. From the figure, we see that*

both schemes with adaptive DFGSC work well in suppressing the remaining ISI. However, DFGSC-CBDFE performs better

Fig. 5. Learning curves of equalizer output MSE for different schemes.

Fig. 6. Learning curves of channel estimate MSE for different schemes.

than DFGSC-LBDFE. This means that the CBDFE is more
effective in ISI suppression. In Fig. 6, we show the MSE of
the channel estimate, i.e., E* {|[h*0(0)

**h**0(1)

*0(γ*

**· · · h***−*1)]

*H*

_{w}*q − wh(k)|*2

*}, for the schemes with the adaptive*

DFGSC. We observe that these MSE values become small when
the decision is reliable (about*−40 dB for DFGSC-CBDFE and*
about*−35 dB for DFGSC-LBDFE). With the accurate channel*
estimate, the DFGSC can eliminate the desired signal from the
error signal, and the CBDFE can construct a better feedback
filter for equalization. This is also where the improvement
comes from.

To show the robustness of the DFGSC against spatial
signa-ture mismatch, we repeat the aforementioned scenario with a
5*◦*, DOA mismatch of the desired signal, i.e., the actual main
DOA is *−5◦*, and the presumed main DOA is 0*◦*. We plot
the resultant GSC output SINR, beam patterns, and equalizer
output MSE in Figs. 7–9, respectively. In Fig. 7, we observe
that the SINR performance gap between DFGSC-CBDFE and
DFGSC-LBDFE becomes larger, and GSC-LBDFE completely
fails due to signal cancellation. In addition, Fig. 8 shows that
DFGSC-CBDFE can keep the beam pattern very close to the

Fig. 7. Learning curves of GSC output SINR for different schemes in which
the actual main DOA is*−5◦*, and the presumed main DOA is 0*◦*.

Fig. 8. Beam patterns of different schemes in Fig. 7 after 5000 iterations in
which the actual main DOA is*−5◦*, and the presumed main DOA is 0*◦*.

Fig. 9. Learning curves of equalizer output MSE for different schemes in
which the actual main DOA is*−5◦*, and the presumed main DOA is 0*◦*.

TABLE II

PARAMETERSUSED INSIMULATIONS OFPARTB: (a) STEPSIZES AND(b) CHANNELSETTINGS

Fig. 10. Learning curves of equalizer output MSE for different schemes with spatial ISI.

optimum. In Fig. 9, we see that DFGSC-CBDFE gives the smallest equalizer output MSE and, thus, performs best. All these indicate the DFGSC is better than the conventional GSC, and the CBDFE is better than the LBDFE.

*B. Channels With Both Temporal and Spatial ISI*

In this section, we consider a more general case where ISI
comes from different directions and time instants. Again, the
CCINR is fixed at 25 dB, and the SNR is 20 dB. The filter
length for the blind DFE is the same as given previously. The
step sizes and channel settings used are shown in Table II. From
the table, we see that the ISI comes from different directions,
and the effect of both coherent interference and angular spread
**is included. Furthermore, the step sizes for w***h* are 1*× 10−3*.

*The parameter ξ is selected as 0.005, for which f (k) is close*
to unity around 1000 iterations. Fig. 10 shows the equalizer
output MSE with 16 quadrature amplitude modulation for
the three schemes. To observe the convergence behavior of
these schemes, we exclude the decision-directed mode and
use the MMA blind algorithm all the way first. We see that
DFGSC-CBDFE outperforms the other two schemes in terms of
both convergence rate and equalizer output MSE. The channel

Fig. 11. SER performance for the scenario in Fig. 10.

Fig. 12. Learning curves of equalizer output MSE for different schemes with spatial ISI (artificial errors added as five consecutive errors from the 2000th iteration, ten interleaved errors from the 3000th iteration, and ten consecutive errors from the 4000th iteration).

estimator in the proposed scheme really helps to construct a more stable and efficient blind DFE. Then, the symbol-error-rate (SER) performance for the three schemes under the same settings is given in Fig. 11. The result is observed under different iteration intervals. Again, DFGSC-CBDFE performs best and offers smaller SER in all cases. Note that GSC-LBDFE fails as well due to signal cancellation.

Finally, we repeat the same experiment with the decision-directed MMSE training in the tracking period (after 1000 itera-tions). However, some artificial errors are added in the decision process. To force an error to occur, we randomly shift the decision to a constellation point near its true value. The errors are added as follows: five consecutive errors from the 2000th iteration, ten interleaved errors from the 3000th iteration, and ten consecutive errors from the 4000th iteration. The result is presented in Fig. 12. Again, it shows that DFGSC-CBDFE performs better than LBDFE. The proposed CBDFE can reconverge after the bursty errors, but

DFGSC-LBDFE diverges. This clearly shows that the CBDFE is less sensitive to decision errors and makes the whole processing scheme more reliable. We then conclude that DFGSC-CBDFE is the best hybrid scheme for the scenario we consider.

VII. CONCLUSION

In this paper, a new adaptive STE has been developed for the
suppression of both CCI and ISI. The proposed STE introduces
a hybrid of an adaptive DFGSC and an adaptive CBDFE.
*With the main DOA known a priori, training sequences are*
not required for the adaptation of the whole STE. We show
that the included channel estimator can not only improve the
performance of the conventional adaptive GSC but also make
the blind DFE more reliable. For spatial processing, the DFGSC
improves the CCI suppression capability when implemented
with the LMS algorithm. In addition, the adaptive DFGSC
with the simple point distortionless constraint is robust against
multipath propagation environments and spatial signature
er-rors. For temporal processing, the proposed CBDFE can have
better performance than the LBDFE. With our special
adapta-tion, the problem of error propagation is reduced. Simulation
results verified our analysis and confirmed that the proposed
STE can achieve good performance, even in severe channel
environments.

REFERENCES

*[1] J. G. Proakis, Digital Communications, 3rd ed. New York: *
McGraw-Hill, 1995.

[2] R. B. Ertel, P. Cardieri, K. W. Sowerby, T. S. Rappaport, and J. H. Reed,
“Overview of spatial channel models for antenna array communication
*systems,” IEEE Pers. Commun., vol. 5, no. 1, pp. 10–22, Feb. 1998.*
*[3] R. A. Monzingo and T. W. Miller, Introduction to Adaptive Arrays.*

New York: Wiley, 1980.

[4] L. C. Godara, “Application of antenna arrays to mobile communications.
*II. Beam-forming and direction-of-arrival considerations,” Proc. IEEE,*
vol. 85, no. 8, pp. 1195–1245, Aug. 1997.

*[5] G. V. Tsoulos, Ed., Adaptive Antennas for Wireless Communications,*
Piscataway, NJ: IEEE Press, 2001.

[6] R. Kohno, “Spatial and temporal communication theory using adaptive
*antenna array,” IEEE Pers. Commun., vol. 5, no. 1, pp. 28–35, Feb. 1998.*
[7] A. J. Paulraj and B. C. Ng, “Space–time modems for wireless
*per-sonal communications,” IEEE Pers. Commun., vol. 5, no. 1, pp. 36–48,*
Feb. 1998.

[8] J.-W. Liang, J.-T. Chen, and A. J. Paulraj, “A two-stage hybrid approach
*for CCI/ISI reduction with space–time processing,” IEEE Commun. Lett.,*
vol. 1, no. 6, pp. 163–165, Nov. 1997.

[9] N. Ishii and R. Kohno, “Spatial and temporal equalization based on
*an adaptive tapped delay line array antenna,” IEICE Trans. Commun.,*
vol. E78-B, no. 8, pp. 1162–1169, Aug. 1995.

[10] M. Chiani and A. Zanella, “Spatial and temporal equalization for
*broad-band wireless indoor networks at millimeter waves,” IEEE J. Sel. Areas*

*Commun., vol. 17, no. 10, pp. 1725–1734, Oct. 1999.*

[11] M.-L. Leou, C.-C. Yeh, and H.-J. Li, “A novel hybrid of adaptive array
*and equalizer for mobile communications,” IEEE Trans. Veh. Technol.,*
vol. 49, no. 1, pp. 1–10, Jan. 2000.

[12] K. Hayashi and S. Hara, “A new spatio-temporal equalization method
*based on estimated channel response,” IEEE Trans. Veh. Technol., vol. 50,*
no. 5, pp. 1250–1259, Sep. 2001.

[13] G. V. Tsoulos, “Smart antennas for mobile communication systems:
*Bene-fits and challenges,” Electron. Commun. Eng. J., vol. 11, no. 2, pp. 84–94,*
Apr. 1999.

*[14] J. C. Liberti and T. S. Rappaport, Smart Antennas for Wireless *

*Commu-nications: IS-95 and Third Generation CDMA Applications. Englewood*

Cliffs, NJ: Prentice-Hall, 1999.

[15] B. Suard, G. Xu, H. Liu, and T. Kailath, “Uplink channel capacity
*of space-division-multiple-access schemes,” IEEE Trans. Inf. Theory,*
vol. 44, no. 4, pp. 1468–1476, Jul. 1998.

[16] G. M. Galvan-Tejada and J. G. Gardiner, “Theoretical model to determine
*the blocking probability for SDMA systems,” IEEE Trans. Veh. Technol.,*
vol. 50, no. 5, pp. 1279–1288, Sep. 2001.

[17] L. J. Griffiths and C. W. Jim, “Alternative approach to linear constrained
*adaptive beamforming,” IEEE Trans. Antennas Propag., vol. AP-30,*
no. 1, pp. 27–34, Jan. 1982.

*[18] S. Haykin, Adaptive Filter Theory, 3rd ed. Upper Saddle River, NJ:*
Prentice-Hall, 1996.

[19] J. Labat, O. Macchi, C. Laot, “Adaptive decision feedback equalization:
*Can you skip the training period?” IEEE Trans. Commun., vol. 46, no. 7,*
pp. 921–930, Jul. 1998.

[20] Y. Lee and W.-R. Wu, “A robust adaptive generalized sidelobe canceller
*with decision feedback,” IEEE Trans. Antennas Propag., vol. 53, no. 11,*
pp. 3822–3832, Nov. 2005.

[21] O. L. Frost, III, “An algorithm for linearly constrained adaptive array
*processing,” Proc. IEEE, vol. 60, no. 8, pp. 926–935, Aug. 1972.*
[22] J. Yang, J. J. Werner, and G. A. Dumont, “The multimodulus blind

*equalization and its generalized algorithms,” IEEE J. Sel. Areas Commun.,*
vol. 20, no. 5, pp. 997–1015, Jun. 2002.

[23] L. M. Garth, “A dynamic convergence analysis of blind
*equaliza-tion algorithms,” IEEE Trans. Commun., vol. 49, no. 4, pp. 624–634,*
Apr. 2001.

[24] P. K. Shukla and L. F. Turner, “Channel-estimation-based adaptive DFE
*for fading multipath radio channels,” Proc. Inst. Electr. Eng. I, vol. 138,*
no. 6, pp. 525–543, Dec. 1991.

[25] A. A. Rontogiannis and K. Berberidis, “Efficient decision feedback
*equal-ization for sparse wireless channels,” IEEE Trans. Wireless Commun.,*
vol. 2, no. 3, pp. 570–581, May 2003.

[26] T.-J. Shan and T. Kailath, “Adaptive beamforming for coherent signals
*and interference,” IEEE Trans. Acoust., Speech, Signal Process.,*
vol. ASSP-33, no. 3, pp. 527–536, Jun. 1985.

[27] S. Valaee, B. Champagne, and P. Kabal, “Parametric localization of
*dis-tributed sources,” IEEE Trans. Signal Process., vol. 43, no. 9, pp. 2144–*
2153, Sep. 1995.

*[28] G. H. Golub and C. F. Van Loan, Matrix Computation, 3rd ed. Baltimore,*
MD: The Johns Hopkins Univ. Press, 1996.

[29] B. Widrow, K. Duvall, R. Gooch, and W. Newman, “Signal cancellation
*phenomena in adaptive antennas: Causes and cures,” IEEE Trans. *

*Anten-nas Propag., vol. AP-30, no. 3, pp. 469–478, May 1982.*

**Yinman Lee (M’06) was born in Hong Kong. He**

received the Ph.D. degree from National Chiao Tung University, Hsinchu, Taiwan, R.O.C., in 2006.

He is currently with the Graduate Institute of Communication Engineering, National Chi Nan University, Nantou, Taiwan. His research interests include adaptive signal processing, wireless commu-nications, and multiple antenna systems.

**Wen-Rong Wu (M’89) received the B.S. degree**

in mechanical engineering from Tatung Institute of Technology, Taipei, Taiwan, R.O.C., in 1980, and the M.S. degrees in mechanical and electrical engineer-ing and the Ph.D. degree in electrical engineerengineer-ing from the State University of New York, Buffalo, in 1985, 1986, and 1989, respectively.

Since August 1989, he has been a faculty mem-ber with the Department of Communication Engi-neering, National Chiao Tung University, Hsinchu, Taiwan. His research interests include statistical sig-nal processing and digital communications.