行政院國家科學委員會專題研究計畫 成果報告
4n 階正交群之廣義 Shalika 模型的分類
研究成果報告(精簡版)
計 畫 類 別 ： 個別型 計 畫 編 號 ： NSC 99-2115-M-006-007- 執 行 期 間 ： 99 年 08 月 01 日至 100 年 10 月 31 日 執 行 單 位 ： 國立成功大學數學系暨應用數學所 計 畫 主 持 人 ： 粘珠鳳 計畫參與人員： 碩士班研究生-兼任助理人員：黎福全 大專生-兼任助理人員：王志文 博士班研究生-兼任助理人員：陳將恩 報 告 附 件 ： 國外研究心得報告 公 開 資 訊 ： 本計畫可公開查詢中 華 民 國 100 年 11 月 08 日
中文摘要： 特殊正交群上廣義夏萊克模型存在的必要性 英文摘要： Let $F$ denote a $p$-adic local field of
characteristic zero.
In this paper, we investigate the structures of irreducible admissible representations of
$\SO_{4n}(F)$ having nonzero generalized Shalika models
and find relations between the generalized Shalika models and the local Arthur parameters, which support our conjectures on the local Arthur parametrization and the local Langlands functoriality in terms of the dual group associated
to the spherical variety, which is attached to the generalized Shalika models.
GENERALIZED SHALIKA MODELS OF p-ADIC SO4n
AND FUNCTORIALITY
DIHUA JIANG, CHUFENG NIEN, AND YUJUN QIN
Abstract. Let F denote a p-adic local field of characteristic zero. In this paper, we investigate the structures of irreducible admissi-ble representations of SO4n(F ) having nonzero generalized Shalika
models and find relations between the generalized Shalika models and the local Arthur parameters, which support our conjectures on the local Arthur parametrization and the local Langlands functori-ality in terms of the dual group associated to the spherical variety, which is attached to the generalized Shalika models.
1. Introduction
This paper is a sequel of our previous work on the characterization of symplectic representations of p-adic group GL2nin terms of the
gen-eralized Shalika models on the F -split special orthogonal group SO4n
([JQ07], [JNQ08], [JNQ10-1], and [JNQ10-2]), where F is a p-adic local field of characteristic zero. For simplicity of notation, we use G for an algebraic group and also for its F -rational points of G. Our objective here is to study the structure of irreducible admissible representations of SO4n, which have a nonzero generalized Shalika model.
Date: June, 2010.
2000 Mathematics Subject Classification. Primary 11F70, 22E50; Secondary 11F85, 22E55.
Key words and phrases. Generalized Shalika models, Representations of p-adic
groups, Arthur parameters, Functoriality.
The work of the first named author is supported in part by NSF (USA) grant DMS-0653742 and DMS-1001672, and by The Chinese Academy of Sciences. The second name author is supported by NSC 99-2115-M-006-007- and NCTS Taiwan. The third named author is supported partly by the program PCSIRT (Program for Changjiang Scholars and Innovative Research Team) in East China Normal University. All three authors are supported in part by NSFC 10701034, P.R.China. The authors would like to thank The Morningside Center of Mathematics and the Institute of Mathematics, CAS, for strong support through the summer program over the years.
Let ν1 = 1 and inductively define (1.1) νn = µ 1 νn−1 ¶ , for n ≥ 2, n ∈ N.
We often use abbreviation ν for νn if there is no confusion for the rank
n. Let SO4n be the even special orthogonal group attached to the
non-degenerate 4n-dimensional quadratic vector space over F with respect to ν4n. That is,
SO4n = {g ∈ GL4n| tg · ν4n· g = ν4n}.
We recall from [JQ07] the definition of the generalized Shalika models on SO4n. Let P2n = M2nV2n be the Siegel parabolic subgroup of SO4n
consisting of elements of the following form:
(1.2) (g, X) = µ g 0 0 g∗ ¶ µ In X In ¶ ,
where g ∈ GL2n and g∗ = ν2n tg−1ν2n, and X satisfiestX = −ν2nXν2n.
The generalized Shalika subgroup H2n of SO4n is the subgroup of
P2n consisting of elements (g, X) with g ∈ Sp2n. Here the symplectic
group is given by Sp2n = {g ∈ GL2n| tg · J2n· g = J2n}, where J2n is given by J2n = µ νn −νn ¶ , n ∈ N.
Define a character ψH of H2n (We write H = H2n if there is no
confu-sion) by letting ψH((g, X)) = ψ(tr(J2nXν2n)) (1.3) = ψ(tr( µ −In In ¶ X)) (1.4)
where ψ is a nontrivial character of F .
The generalized Shalika functional or ψH-functional of an irreducible
admissible representation (σ, Vσ) of SO4n is a nonzero functional in the
following space
HomH(Vσ, ψH) ∼= HomSO4n(Vσ, Ind SO4n
H (ψH)).
Here for any closed subgroup H of G, and for any admissible represen-tation τ of p-adic H, IndGH(τ ) denotes the normalized smooth induction, and indGH(τ ) denotes the normalized compact induction.
The local uniqueness of the generalized Shalika models is proved by Nien in [N10], that is, for any irreducible admissible representation (σ, Vσ) of SO4n, the dimension of the space
HomH(Vσ, ψH)
is at most one. Let `ψH be a nonzero ψH-functional of an irreducible
admissible representation (σ, Vσ) of SO4n. For v ∈ Vσ, and g ∈ SO4n,
we define
HψH(g, v) := `ψH(σ(g)(v)),
which is a ψH-generalized Shalika function on SO4nattached to v. The
space consisting of all ψH-generalized Shalika functions with v ∈ Vσ is
called the ψH-generalized Shalika model of σ or just the ψH-model of
σ.
One of the key local results of [JQ07] is that the Shalika models (which is recalled below) and the generalized Shalika models are in-trinsically related through the parabolic induction.
For an irreducible, unitary, supercuspidal representation (τ, Vτ) of
GL2n, we consider the following unitarily induced representation of
SO4n
ISO4n(s, τ ) = IndSO4n
P2n (| det |
s
2 · τ )
which consisting of all smooth Vτ-valued functions φτ,s on SO4n, such
that φτ,s(m(a)ng) = | det a| s 2+ 2n−1 2 τ (a)φτ,s(g), where m(a) ∈ M2n with a ∈ GL2n, n ∈ V2n.
Theorem 1.1 (Theorem 3.1, [JQ07]). The unitarily induced represen-tation ISO4n(s, τ ) admits a nonzero generalized Shalika functionals only when s = 1. In that case, ISO4n(1, τ ) admits a nonzero generalized Sha-lika functional if and only if the supercuspidal datum τ admits a nonzero Shalika functional. The generalized Shalika functionals of ISO4n(1, τ ) is unique up to scalar, and if nonzero, they must factor through the unique Langlands quotient LSO4n(1, τ ).
We recall from [JS90] the definition of the Shalika models for GL2n.
Take the maximal parabolic subgroup Pn,n = Mn,nNn,n of GL2n with
Mn,n= GLn× GLn, and Nn,n = {n(X) = µ In X 0 In ¶ ∈ GL2n}. Define a character ψNn,n(n(X)) = ψ(tr(X)). 3
The stabilizer of ψNn,n in Mn,n is GL
∆
n, the diagonal embedding of GLn
into Mn,n. The Shalika subgroup Sn is defined to be
(1.5) Sn = {s(a, X) = diag(a, a)n(X)|a ∈ GLn, n(X) ∈ Nn,n}.
It is clear that Sn = GL∆nnNn,n, a semi-direct product with the radical
Nn,n being a normal subgroup. Denote by ψSn the extension of ψNn,n
from Nn,n to the Shalika subgroup Sn, such that ψSn is trivial on GL
∆
n.
The Shalika functionals of an irreducible admissible representation (τ, Vτ) of GL2n are nonzero elements of the following space
HomSn(Vτ, ψSn) ∼= HomGL2n(Vτ, Ind GL2n
Sn (ψSn)).
Any nonzero Shalika functional `ψ in HomSn(Vτ, ψSn) gives rise to an
embedding of Vτ into the full induction IndGLSn2n(ψSn), the image of
which is called a local Shalika model of Vτ. The local uniqueness of
Shalika models was first proved by Jacquet-Rallis in [JR96] and then by Nien in [N09-2] by using a different argument.
The first result of this paper is the following theorem:
Theorem 1.2. Let (σ, Vσ) be an irreducible admissible representation
of SO4n. Assume that Vσ has a nonzero generalized Shalika model (or
ψH-model). Then there exists an irreducible admissible representation
τ of GL2n such that Vσ is a quotient of the following induced
represen-tation
IndSO4n
P2n (τ | det | 1 2).
The idea of the proof goes as follows. It is well-known ( a version of the Jacquet submodule theorem) that any irreducible admissible rep-resentation (σ, Vσ) of SO4n can be realized as a quotient of unitarily
induced representation IndSO4n
Q (π). This means that there is a
surjec-tive SO4n-equivariant mapping
IndSO4n
Q (π) → Vσ → 0,
where Q is a standard parabolic subgroup of SO4n with its Levi
sub-group isomorphic to
GLn1 × · · · × GLnr × SO4n0 and the representation π can be expressed as
π = τ1| det |s1 ⊗ · · · ⊗ τr| det |sr ⊗ σ0
with τi supercuspidal representation of GLni, i = 1, · · · , r and σ0
su-percuspidal representation of SO4n0.
If Vσ has a nonzero ψH-functional, then IndSOQ 4n(π) has a nonzero ψH
-functional. We will show in Proposition 2.2 that if n0 is nonzero, then 4
the irreducible supercuspidal representation σ0 of SO4n0 has a nonzero generalized Shalika model on SO4n0.
Ginzburg proved in [G95] (Lemma 5.1) that any cuspidal automor-phic form on GSO4n has no nonzero generalized Shalika period
(al-though this terminology was not used in [G95]). From the proof of Lemma 5.1 of [G95], it is not hard to see that the proof works also for cuspidal automorphic forms on SO4n. Furthermore, the local analogy
of the proof shows that any irreducible supercuspidal representation σ of SO4n has no nonzero generalized Shalika models. This can also
be proved by using the globalization argument of Prasad and Schulze-Pillot (Theorem 4.1, [PSP08]). Hence we obtain that n0 must be zero
if Vσ has a nonzero generalized Shalika model.
It follows that if Vσ has a nonzero generalized Shalika model, then
we must have
IndSO4n
Q (π) → Vσ → 0,
where Q is a standard parabolic subgroup of SO4n with its Levi
sub-group isomorphic to
GLn1 × · · · × GLnr
and the representation π can be expressed as π = τ1| det |s1 ⊗ · · · ⊗ τr| det |sr
with τ1, · · · , τr all supercuspidal. Finally we write IndSOQ 4n(π) as
IndSO4n P2n ((Ind GL2n Pn1,··· ,nr(τ1| det |s 0 1 ⊗ · · · ⊗ τ r| det |s 0 r))| det |12). Hence there is an irreducible quotient τ of
IndGL2n Pn1,··· ,nr(τ1| det |s 0 1 ⊗ · · · ⊗ τ r| det |s 0 r) such that (1.6) IndSO4n P2n (τ | det | 1 2) → Vσ → 0.
Then for r = 1, Theorem 1.2 follows from Theorem 1.1. For r > 1, Theorem 1.2 will be established after we verify Proposition 2.2.
By comparing Theorem 1.2 to Theorem 1.1, we expect to produce a certain model for the irreducible admissible representation τ of GL2n
such that (1.6) holds with Vσ having a nonzero generalized Shalika
model. To this end, we introduce the following new family of models. For an integer 0 ≤ r ≤ n, in the standard maximal parabolic sub-group
P2r,2n−2r = M2r,2n−2rN2r,2n−2r = (GL2r× GL2n−2r)N2r,2n−2r
of GL2n, we define a subgroup Nr of GL2n by
(1.7) Nr:= (Sr× Sp2n−2r)N2r,2n−2r, 5
where Sr is the Shalika subgroup of GL2r as defined in (1.5), and the
symplectic group Sp2n−2r is embedded into GL2n−2r naturally. We
de-note by n(s, h, x) the elements of Nrwith s = s(a, X) ∈ Sr, h ∈ Sp2n−2r
and x ∈ N2r,2n−2r, and define a 1-dimensional representation θNr of Nr
by
(1.8) θNr(n(s, h, x)) := ψSr(s)| det a|
n−r.
It is easy to see from the definition of Nr that θNr is well defined.
For any irreducible admissible representation (τ, Vτ) of GL2n, if the
following space
(1.9) HomNr(Vτ, θNr) 6= 0,
we say that the representation Vτ has a nonzero θNr-functional or a
nonzero θNr-model.
It is clear that when r = 0, N0 = Sp2n. Hence θN0-model is the symplectic model of GL2n, which was first studied by Klyachko in 1984
over a finite field and by Heumos and Rallis in 1990 over a p-adic local field ([HR90]). On the other hand, when r = n, Nn = Sn. Hence
θNn-model recovers the Shalika model of GL2n, which was first used by
Jacquet and Shalika in 1990 ([JS90]). In this sense, we may view this new family of models as an interpolation between the symplectic model and the Shalika model for GL2n. It should be very interesting to study
further properties of this new family of models. We will consider this in other occasion.
The second result of this paper is
Theorem 1.3. Let τ be an irreducible admissible representation of GL2n. If the induced representation
IndSO4n
P2n (τ | det | 1 2)
has a nonzero generalized Shalika model, then τ has a nonzero θNr
-model for some integer 0 ≤ r ≤ n.
We note that Theorem 1.3 is to extend Theorem 1.1 (Theorem 3.1 in [JQ07]) to great generality. However, the statement in Theorem 1.3 is not as complete as that in Theorem 1.1. The main reason is that for the moment, we only have very limited knowledge about the family of θNr-models with 0 ≤ r ≤ n. We will come back to this
issue at the end of §3 after we finish the proof of Theorem 1.3, since the geometric structures occurring in the proof yield more information about these models. We also remark that the proof of Theorem 1.3 looks close to that of Theorem 1.1, but it is much more technical since it needs more complete geometric structures of the H-orbits on the
generalized flag variety P2n\SO4n over F . See §3.1 for the details. We
also discuss the existence and the local uniqueness issues in §3.2. This discussion reduces the issues to the disjointness of the θNr-models and
the Klyachko models ([HR90], [O06], [OS07], [OS08-1], [OS08-2], and [N09-1]), which is given in §5.
In §4, we will explain our main results in terms of the local Arthur parametrization of irreducible admissible representations of SO4n, and
state our conjecture (Conjecture 4.1) on the local Langlands functo-riality in terms of the dual group associated to the spherical variety, which gives the generalized Shalika models. Based on this, we prove Theorem 4.2, which yields the relation between the local generalized Shalika models on SO4n and the local Arthur parametrization, and
state Conjecture 4.3, which characterizes the symplectic type of all ir-reducible admissible representations of GL2nin terms of the θNr-models
with r = 0, 1, 2, · · · , n. The special cases of this conjecture were es-tablished in [JNQ08] and in [OS07] for different types of irreducible admissible representations of GL2n. We remark that the discussion in
the section was motivated by Y. Sakellaridis’ wonderful lecture in the workshop on Relative trace formula and periods of automorphic forms at The American Institute of Mathematics, 2009, announcing his joint work with A.Venkatesh on periods and harmonic analysis on spherical varieties. In a sense, the conjectures and the results discussed in Sec-tion 4 support their more general conjectures on Plancherel formula on spherical varieties.
Finally, we would like to thank the referee for helpful comments.
2. Proof of Theorem 1.2
We start with the following statement for supercuspidal representa-tions of SO4n.
Theorem 2.1 (Local Version of Lemma 5.1, [G95]). Let ρ be an irre-ducible supercuspidal representation of SO4n. Then ρ does not have a
nontrivial generalized Shalika model.
Ginzburg proved in [G95] (Lemma 5.1) that any cuspidal automor-phic form on GSO4n has no nonzero generalized Shalika period
(al-though this terminology was not used in [G95]). From the proof of Lemma 5.1 of [G95], it is not hard to see that the proof works also for cuspidal automorphic forms on SO4n. Furthermore, the local analogy
of the proof shows that any irreducible supercuspidal representation σ of SO4n has no nonzero generalized Shalika module. This can also
be proved by using the globalization argument of Prasad and Schulze-Pillot (Theorem 4.1, [PSP08]). We omit the details without repeating the same argument here.
2.1. Proof of Theorem 1.2. Let r + m = 2n, denote by (2.1) Pr,2m = (GLr× SO2m)Nr,2m
the standard parabolic subgroup of SO4n. When r = 2n, Pr,2m = P2n is
the Siegel parabolic subgroup of SO4n. When m = 2n, Pr,2m = SO4n.
To prove Theorem 1.2, it is enough to prove the following Proposi-tion:
Proposition 2.2. Let Pr,2m with r + m = 2n be the standard maximal
parabolic subgroups of SO4n, τ be a smooth representation of GLr of
finite length, and ρ be a supercuspidal representation of SO2m. If m ≥ 1,
then the induced representation π = IndSO4n
Pr,2m(τ ⊗ ρ)
does not admit a generalized Shalika model of SO4n.
First we prove a special case of Theorem 1.2.
Lemma 2.3. Let π be an irreducible representation of SO4 admitting
a nontrivial generalized Shalika model, then π must be a subquotient of IndSO4
P2 (τ ), for some representation τ of GL2 admitting either a non-trivial Shalika model or a symplectic model.
Proof. By classification, π must be either supercuspidal or a subquo-tient of IndSO4
Pr,4−2r(τr⊗ σ4−2r) with 1 ≤ r ≤ 2, where τr is an irreducible
representation of GLr and σ is an irreducible supercuspidal
represen-tation of SO4−2r.
(1) If π is supercuspidal, then by Theorem 2.1 it does not admit a nontrivial generalized Shalika model.
(2) Since SO2 ∼= F∗ , the induced representation IndSOP1,24(τ1⊗σ2) can be viewed as IndSO4
P2 (τ ) for some τ (not necessarily irreducible) representation of GL2. The fact that τ must admit either a
nontrivial Shalika model or a symplectic model follows from Theorem 1.3 or from a direct and easy computation.
¤ The proof of Proposition 2.2 will be given in the next subsection (2.2). Here we set up notation and auxiliary properties. Let ±ei ±
ej denote the roots as convention. We also denote by Uα the one
parameter subgroup of SO4n corresponding to the root α. 8
Proposition 2.4. Let W1 be the set consisting of elements w ∈ W (SO4n) satisfying (2.2) w(ek) = ek, if k ≤ tw; w(ek) = −ek, if tw < k ≤ r; w(ek) = −ek, if r < k ≤ t0w; w(ek) = ek, if t0w < k ≤ 2n,
for some 0 ≤ tw ≤ r ≤ t0w ≤ 2n and t0w − tw is even. Let S1 be a
complete set of representatives for [GL2n/Sp2n]4 Then
Pr,2m\SO4n/H2n = ∪w∈W1,g∈S1Pr,2mwgH2n.
Proof. The result follows that W1 is a complete set of representatives
for
Pr,2m\SO4n/P2n
and
P2n = [GL2n/Sp2n]4· H2n.
¤ In order to prove Proposition 2.2, it is enough to show the following:
HomSO4n(Ind SO4n
Pr,2m(τ ⊗ ρ), Ind
SO4n
H (ψH)) = {0}.
Here H = H2n and τ ⊗ ρ also denotes its extension to Pr,2m, which is
trivial on the unipotent radical of Pr,2m. We also use the same
identi-fication for other inducing data in parabolic induction. By reciprocity law, HomSO4n(Ind SO4n Pr,2mτ ⊗ ρ, Ind SO4n H ψH) ∼= HomH(IndSOPr,2m4n(τ ⊗ ρ)|H, ψH).
Notice that H is unimodular. Let S denote a complete set of represen-tatives of Pr,2m\SO4n/H. Let δG be the modular function of a group
G. Then up to semisimplification, we have as vector spaces: IndSO4n Pr,2m(τ ⊗ ρ)|H ∼= ⊕g∈Sind H Hg(τ ⊗ ρ ⊗ δ 1 2 Pr,2m) g where Hg = g−1P
r,2mg ∩ H, and the representation (τ ⊗ ρ ⊗ δ
1 2 Pr,2m) g acts on Vτ ⊗ρ by (τ ⊗ ρ ⊗ δ12 Pr,2m) g(h) = (τ ⊗ ρ ⊗ δ12 Pr,2m)(ghg −1), for h ∈ Hg.
Definition 2.5. Let τ be a representation of GLr and ρ be a
repre-sentation of SO2m. We say that a double coset Pr,2mgH, g ∈ GL2n,
is admissible for representation π = IndSO4n
Pr,2m(τ ⊗ ρ) if the following inequality holds HomHg((τ ⊗ ρ ⊗ δ 1 2 Pr,2m) g, ψ HδH−1g) 6= {0}, 9
where ψH is identified with its restriction to Hg.
More generally, let G be a group and Hi its subgroups, i = 1, 2.
Assume that H2 is unimodular. Let σi be a representation of Hi. We
say that a double coset H1gH2, g ∈ G, is admissible between (σ1, H1)
and (σ2, H2) if the following inequality holds
HomHg((σ1⊗ δ 1 2 H1) g, σ 2⊗ δ−1Hg) 6= {0}, where Hg = g−1H
1g ∩ H2, and the representation (σ1 ⊗ δ 1 2 H1) g acts on Vσ1 by (σ1⊗ δ 1 2 H1) g(h) = (σ 1⊗ δ 1 2 H1)(ghg −1), for h ∈ Hg.
To show Proposition 2.2, now it reduces to show that dim HomHg((τ ⊗ ρ ⊗ δ 1 2 Pr,2m) g, ψ HδH−1g) = 0.
for all representatives g ∈ S. That is to show that there is no admissible cosets Pr,2mgH for all representatives g. From Proposition 2.4, we need
to consider the admissibility of Pr,2mwgH, with w ∈ W1, and g ∈ S1.
To simplify the calculation of admissibility of double cosets, we ex-tend the notion of generalized Shalika groups as follows. Let A be a nonsingular skew symmetric matrix of degree 2n. Define
HA= Sp2n(A)V2n,
where elements in HAare (h, X) ∈ P2n(notations as in Eq. (1.2)) with
h ∈ Sp2n(A), and
Sp2n(A) = {x ∈ GL2n|txAx = A}.
Let g ∈ GL2n satisfy tgAg = J2n. Then Sp2n(A) = gSp2n(J2n)g−1,
and gHg−1 = H
A. Define a character ψHA on HA by
(2.3) ψHA(h, X) = ψ(tr(AXν2n)), (h, X) ∈ HA.
It is well defined. (Refer to [JNQ10-1].) When A = J2n, HA = H is
the generalized Shalika group as defined before, and ψHA = ψH.
Definition 2.6. For any skew symmetric matrix A and w ∈ W1, we
say that a double coset Pr,2mwHA is admissible for representation
π = IndSO4n
Pr,2m(τ ⊗ ρ) if the following inequality holds
HomHw,A((τ ⊗ ρ ⊗ δ 1 2 Pr,2m) w, ψ HAδ −1 Hw,A) 6= {0}, where Hw,A = w−1P r,2mw ∩ HA.
Lemma 2.7. The double coset Pr,2mwgHA with w ∈ W1, g ∈ GL2n is
admissible if and only if Pr,2mwHB is admissible where B =tg−1Ag−1.
Let Uk denote the upper triangular maximal unipotent subgroup of
GLk and W (GLk) the Weyl group of GLk, identified with the group
of permutation matrices in GLk. The following lemma gives a nice
characterization of nonsingular skew symmetric matrices.
Lemma 2.8 ([JR92], Lemma 2). Every nonsingular skew symmetric matrix of degree 2n can be written in the form
s = uσλ ut
with u ∈ U2n, λ is a diagonal matrix in GL2n, and σ ∈ W (GL2n) such
that
(2.4) σ2 = 1, σλσ−1 = −λ.
Let A be a subset of GL2n defined by
A = {σλ | λ is diagonal , σ ∈ W (GL2n), σ2 = 1, σλσ−1 = −λ}.
Then A consists of nonsingular skew symmetric matrices with one and only one nonzero element at each row and column, which is called monomial. Therefore, by Lemma 2.7 and 2.8, to prove Proposition 2.2, it suffices to show that the coset Pr,2mwuHA is not admissible for any
w ∈ W1, u ∈ U2n, A ∈ A, and u ∈ U2n (identified with its diagonal
embedding in SO4n).
2.2. Proof of Proposition 2.2. The case of m = 2n is shown in Theorem 2.1. Now we assume that m < 2n, and want to show that Pr,2mwuHA is not admissible for any fixed w ∈ W1, u ∈ U2n, and
A ∈ A. We will proceed by induction on n and we assume that Pr0,2m0wuHA ⊂ SO4q
for all r0+ m0 = 2q, q < n is not admissible with respect to any
π0 = IndSO4q
Pr0,2m0(τ
0⊗ ρ0),
where τ0 is any representation (not necessarily irreducible ) of GL r0
and ρ0 is an irreducible supercuspidal representation of SO
2m0. Note
that the case of SO4 (i.e. n = 1) has been treated in Lemma 2.3.
Write A = (ai,j) and define a permutation ι = ιAon [2n] with respect
to A such that as,ι(s) 6= 0. It is well-defined since A is monomial and
ι2 = id since A is skew symmetric. For any skew symmetric matrix T
(not necessary in A), write T = (ti,j), 1 ≤ i, j ≤ 2n. If tk,s 6= 0, then
ψHT|Uα is not trivial, where α is the root corresponding to (k,
4n+1−s)-entry in SO4n. Especially, for A ∈ A,
ψHA|Uβ is not trivial,
where β is the root corresponding to (s, 4n + 1 − ιA(s))-entry in SO4n. 11
Given any u ∈ U2n and A ∈ A, to show that Pr,2mwuHA is not
admissible is equivalent to show that Pr,2mwHBis not admissible, where
B =t u−1Au−1. Write A = µ A1 A2 A3 A4 ¶ and B = µ B1 B2 B3 B4 ¶ ,
where A1, B1 ∈ Matr and A4, B4 ∈ Matm. For all skew symmetric
matrix T = (ti,j), denote
(2.5) CT = {k ≤ r| tk,s 6= 0 for some s ≥ r + 1}.
This means that CT is the set of row indices corresponding to nonzero
rows of T2, when T = µ T1 T2 T3 T4 ¶ , with T1 ∈ Matr, T4 ∈ Matm.
Lemma 2.9. Assume that CBis nonempty (i.e. B2 6= 0). If Pr,2mwHB
is admissible, then w(ek) = −ek for k ∈ CB.
Proof. Assume on the contrary that w(ek) = ek and bk,s 6= 0 for some
s > r and k ∈ CB. For λ ∈ F , take
X(λ) = λ(es+ ek) ∈ Ues+ek.
Since w(es + ek) = ek ± es, the element w−1(X(λ))w belongs to the
unipotent radical of Pr,2m. Hence we have
ψHB(X(λ)) = ψ(±bk,sλ)
(τ ⊗ ρ)(w−1(X(λ))w) = 1,
where the last equality means that the representation τ ⊗ρ acts trivially on elements of the form w−1(X(λ))w and similar expression will also
be used in later paragraphs. Therefore Pr,2mwHB is not an admissible
coset. ¤
Since B = tu−1Au−1, if the k-th row of A
2 is nonzero, then so is
the k-th row of B2. By Lemma 2.9, if PrwuHA is admissible (which is
equivalent to PrwHB is admissible) and CA is nonempty, then w(ek) =
−ek for all k ∈ CA.
Write u ∈ U2n as u = u1 uu24 uu35 u6 , 12
where u1 ∈ Utw, u4 ∈ Ur−tw, u6 ∈ Um, and tw is defined in Equation
(2.2). For g ∈ GL2n, denote the embedding of elements of GL2n into
SO4n by m(g) = diag(g, g∗) ∈ SO 4n. Since m( u1 uu24 u03 u6 ) ∈ w−1P r,2mw,
we can even take
(2.6) u = Itw Ir−t0w u05 Im as the representative in the coset Pr,2mwuHA.
Lemma 2.10. Assume that Pr,2mwuHA is admissible, with u of the
form in (2.6). If s ≤ tw, then tw < ιs ≤ r.
Proof. By Lemma 2.9, ιs ≤ r. Assume on the contrary that ιs ≤ tw.
Take X(λ) = λ(es+ eιs) with λ ∈ F∗. Then wuX(λ)u−1w−1 = X(λ)
belongs to the unipotent radical of Pr and
ψHA(X(λ)) = ψ(±as,ιsλ),
which is not trivial for suitable chosen λ. It contradicts to the admis-sibility of Pr,2mwuHA and hence tw < ιs≤ r. ¤
Lemma 2.11. If tw 6= 0, CB 6= ∅ and w(ek) = −ek, for all k ∈ CB,
where B =tu−1Au−1 with u as in (2.6), then P
r,2mwHB is not
admis-sible.
Proof. Assume that Pr,2mwHB is an admissible coset. Let A01 ∈ Mat2tw
be the left-upper block of A. By Lemma 2.10, we can write A0
1 in the following form A0 1 = µ 0tw C −Ct D ¶ .
Moreover, with the help of some diagonal matrix and permutation ma-trix on {ei|tw ≤ i ≤ r}, which are in w−1Pr,2mw and commute with the
set of unipotent matrix of the form (2.6), we may assume that A01 = µ νtw −νtw ¶ . 13
Let u = Itw 0 0 0 Itw 0 z1 Ir−2tw z2 Im , then B = tu−1Au−1 = 0 νtw 0 −νtwz1 −νtw 0 0 0 D1 D2 (νtwz1)t D3 D4. By Lemma 2.9, −νtwz1 = 0 and B is of the form
(2.7) B = µ 0 νtw −νtw 0 ¶ B0 4 , where B0
4 ∈ Mat2n−2tw is skew symmetric of even rank. Since B is the
form of Eq. (2.7) and CB 6= ∅, by Lemma 2.9 and Eq. (2.2),
r0 := r − 2tw > 0.
For tw 6= 0 (i.e. B10 is nontrivial), the admissibility(Refer to 2.6 for the
definition.) of Pr,2mwHB with respect to π gives:
HomHw,B((τ ⊗ ρ ⊗ δ 1 2 Pr,2m) w, ψ HBδ −1 Hw,B) 6= {0}, where Hw,B = w−1P r,2mw ∩ HB, which is equivalent to HomHw,B((τ ⊗ ρ ⊗ δ 1 2 Pr,2mδHw,B) w, ψ HB) 6= {0},
where Hw,B = Pr,2m∩ wHBw−1. That is there exist a nontrivial
T : Vτ ⊗ρ 7→ C,
such that
(2.8) T (τ ⊗ ρ ⊗ δ12
Pr,2mδHw,B(x) · v) = ψHB(w
−1xw)T (v),
for x ∈ Hw,B, v ∈ Vτ ⊗ρ. Let w0 = w|SO4n−4tw, where w0 and SO4n−4tw
are identified with their embedding in the middle part of SO4n. More
precisely, w = Itw νtw w0 νtw Itw , 14
and Pr0,2m is identified with I2tw Pr0,2m I2tw ⊂ Pr,2m.
Consider Eq. (2.8) for y ∈ Hw0,B0
4 = w0−1Pr0,2mw0∩ HB0 4. Then HomHw0,B04((τ ⊗ ρ ⊗ δ12 Pr,2mδHw,B) w0 , ψHB04) 6= {0}, where Hw0,B0 4 = Pr0,2m∩ w 0H B0 4w 0−1. Hence HomHw0,B04((τ0⊗ ρ0⊗ δ12 Pr0,2mδHw0,B04) w0 , ψHB04) 6= {0}, where τ0 = τ ⊗ δ12 Pr,2mδHw,Bδ −12 Pr0,2mδ −1 H w0,B04|GLr0, ρ0 = ρ ⊗ δ12 Pr,2mδHw,Bδ −12 Pr0,2mδ −1 H w0,B04|SO2m. That is Pr0,2mw0HB0
4 is admissible with respect to π
0 = IndSO4n−2tw
pr0,2m τ
0⊗ρ0.
By Lemma 2.8, there exists u0 ∈ U
4n−4tw and A0 ∈ A(GL2n−2tw) such
that B = tu0−1A0u0−1. By Lemma 2.7, the admissiblity with respect
to π0 of P
r0,2mw0HB0
4 is the same as the admissiblity of Pr0,2mw
0u0H A0,
which contradicts to the induction assumption at the beginning of the subsection 2.2, with 4n − 4tw < 4n.
¤ Lemma 2.12. Assume that tw = 0, then Pr,2mwuHA is not admissible
for all A ∈ A and u ∈ U2n.
Proof. Let ¯U2n denote the opposite unipotent radical of U2n. When
tw = 0, the admissibility of the double coset Pr,2mwuHA is equivalent
to the admissibility of Pr,2mw¯uHA0 for some A0 ∈ A and ¯u ∈ ¯U2n. Since
¯
U2n ⊂ w−1Pr,2mw, Pr,2mw¯uHA0 = Pr,2mwHA0. Now it leaves us to show
the non-admissibility of Pr,2mwHA0, which will be established in the
following Lemmas 2.13 and 2.14. ¤
Lemma 2.13. If CA 6= ∅ and w(ek) = −ek, for all k ∈ CA. Then
Pr,2mwHA is not admissible.
Proof. Let
DA= {t ≥ r + 1| ι(t) ≥ r + 1},
and q be the cardinality of DA (q may equal to zero). Then the
car-dinality of CA = m − q. For k ∈ CA, t ∈ DA, for all α ∈ F∗, choose
β(α) ∈ F∗ such that
gk,t(α) = I2n+ αEι(k),t+ β(α)Eι(t),k ∈ SpA,
where Ei,j = (ek,l) denotes the elementary matrix with its (k, l)-entry ek,l = δk,iδl,j. Define Xk,t(λ) := λ(Eι(k),2n+1−t− Et,2n+1−ι(k)), for λ ∈ F, and Yk,k0(η) := η(Eι(k),2n+1−k0 − Ek0,2n+1−ι(k)), where η ∈ F, k 6= k0 ∈ C A. Then we have (gk,t(α), Xk,t)(I2n, Yk,k0(λ)) ∈ HA∩ w−1Pr,2mw and ψHA((gk,t(α), Xk,t(λ))(I2n, Yk,k0(λ))) = 1. If w0 = w|
SO2m is an even permutation, then according to admissibil-ity, ρ|N0 = 1, where N0 is isomorphic to the unipotent radical N2m−2q,q
of SO2m. If w0 = w|SO2m is an odd permutation, then ˜ρ ∼= ρ0(Refer to Lemma 4.2, [N10]), where the representation space of ρ0 is the same as
ρ such that
ρ0(g) = ρ(w0gw0−1), for g ∈ SO
2m.
Note that if ρ is supercuspidal then so is its contragradient ˜ρ. Hence the admissibility implies ρ0|
N0 = 1, where N0is isomorphic to the unipotent
radical N2m−2q,q of SO2m. Therefore, both of the cases contradict to
the supercuspidality of ρ. Hence Pr,2mwHA can not be admissible. ¤
Lemma 2.14. Let B = µ
B1
B4
¶
, for some skew symmetric matrices B1 ∈ Matr and B4 ∈ Matm, where m and r are both even and nonzero.
Then Pr,2mwHB is not admissible.
Proof. Assume that Pr,2mwHB is admissible. Note that the SO2m part
is invariant under the conjugate action of w ∈ W1 and only regards
to the skew symmetric form B4. According to whether w|SO2m is even or odd, the admissibility of the coset implies that either ρ or ˜ρ has a generalized Shalika model of SO2m. If ρ is supercuspidal, then so is its
contragradient ˜ρ. (Refer to [N10].) Either of the cases contradicts to
Theorem 2.1. ¤
Lemmas 2.9 through 2.14 complete the proof of Proposition 2.2. 3. Proof of Theorem 1.3
In this section, we fix any n ∈ N, and write P = P2n the Siegel
parabolic subgroup of SO4n. We first prove Theorem 1.3 and then
dis-cuss the conditions for the existence and uniqueness of the generalized Shalika functionals on such an induced representation of SO4n.
3.1. Proof of Theorem 1.3. We want to show that if the induced representation π := IndSO4n P (τ | det | 1 2)
for some irreducible admissible representation τ of GL2n, has a nonzero
generalized Shalika model, then τ admits a nonzero θNr-model of GL2n
for some 0 ≤ r ≤ n.
To compare with Theorem 1.1, we consider first the general case when
Σ := IndSO4n
P (τ | det |
s
2).
Following the standard argument, we have to consider the H-orbit de-composition of the generalized flag variety P \SO4n and consider the
admissibility of the H-orbits, from which we deduce the property of τ as stated in Theorem 1.3.
Define certain representatives of Weyl group elements of SO4n:
wi := I2i I2n−2i I2n−2i I2i , 0 ≤ i ≤ n.
Then the set {wi|0 ≤ i ≤ n} is a complete set of representatives for
the generalized Bruhat decomposition P \SO4n/P
of SO4n with respect to the parabolic subgroup P . Hence a complete
set of the representatives of double coset decomposition P \SO4n/H
consists of elements of type wih with 0 ≤ i ≤ n and certain h ∈ GL2n.
Now we consider all double cosets in the form of P wihH, where
h ∈ GL2n, 0 ≤ i ≤ n. By the same argument as in the previous section,
it suffices to consider all double cosets in the form of P wiuH¯ A,
with ¯u ∈ ¯U2n, A ∈ A, where we use ¯u ∈ ¯U2n (the radical of the opposite
of P ) to simplify the computation.
To figure out the admissibility of the cosets P wiuH¯ A with A ∈ A,
we have the following cases: i = 0 and 1 ≤ i ≤ n.
When i = 0, we have the coset P w0uH¯ A= P HA, for all ¯u ∈ ¯U2n, A ∈
A. For any (I2n, X) ∈ V2n, then (I2n, X) ∈ P ∩ HA. On one hand we
have
(τ · | det |s2)((I2n, X)) = 1 17
and on the other hand the character ψHA((I2n, X)) = ψ(trAXν2n) is
non-trivial. Hence P w0uH¯ A is not an admissible coset.
When i = k with 1 ≤ k ≤ n, we consider the double coset P wiuH¯ A.
Define p = G B X Y C D Z W D0 B0 C0 G0 ∈ P, where G, G0 ∈ Mat
2k and D, D0 ∈ Mat2n−2k. Then
wkpw−1k = G0 C0 W D Z C B0 D0 Y B X G . Since w−1
k P wk = (¯uwk)−1P wku for all ¯¯ u ∈ ¯U2n, it suffices to consider
the admissibility of P wkHA for A ∈ A.
Write A := µ A1 A2 A3 A4 ¶ ,
with A1 ∈ Mat2k and A4 ∈ Mat2n−2k. Define t := (I2n, T ) ∈ V2n with
T = µ 0 0 Z 0 ¶ , Z ∈ Mat2n−2k.
Then t ∈ w−1k P wk∩ H. Now on the left hand side, we have
(τ · | det |s2)(t) = 1, and on the right hand side we have
ψHA(t) = ψ(trA4Zν2n−2k),
which is the key condition for us to determine the admissibility of the cosets P wkHA for A ∈ A and k = 1, 2, · · · , n.
For 2k < n, since A is invertible, A4 6= 0. Hence the cosets P wkHA
are not admissible, and when 2k ≥ n and A4 6= 0, the cosets P wkHA
are also not admissible. It remains to consider the cosets P wkHA with
the conditions that 2k ≥ n and A4 = 0.
In this case, we may re-write the 2n × 2n-matrix A as follows: A = aa14 aa25 aa36 a7 a8 0 18
with a1 ∈ Mat2n−2k and a5 ∈ Mat4k−2n. Since A is monomial, there
exists a suitable permutation matrix B = µ B1 B2 ¶ , B1 ∈ Mat2k, B2 ∈ Mat2n−2k
and a diagonal matrix d = diag(d1, · · · , dn, dn, · · · , d1) such that
(dB)A(dB)t= J.
Notice
(wkdB)−1P wkdB = (wk)−1P wk.
Hence P wkHA = P wkH.
In order to consider the admissibility of the cosets P wkH with the
condition that 2k ≥ n, we consider the stabilizer of the coset in P , which is P ∩ wkHwk−1. The elements of the stabilizer can be written as
µ g X g∗ ¶ ∈ P ∩ wkHw−1k , with g = GG13 D R1 R2 D , X = WY Z Y∗ W∗ , where the matrices satisfy the conditions
• G1 ∈ Sp4k−2n, and G3, R1, and R2 are arbitrary,
• D ∈ GL2n−2k and D∗ = ν tD−1ν,
• W ∈ Mat4k−2n,2n−2k and W∗ = ν2n−2kWtν4k−2n,
• Y, Z ∈ Mat2n−2k such that Zt = ν2n−2kZν2n−2k and Y∗ =
ν2n−2kYtν2n−2k, such that (3.1) h = D ∗ W G1 Y G3 D ∈ Sp2n.
Clearly, the subgroup of GL2n consisting of all elements g above is
isomorphic to the group N2n−2k (with 2k ≥ n) as defined in (1.7),
which defines the θN2n−2k-model for GL2n as in (1.9). More explicitly, we write
µ g X g∗ ¶ = G1 W G3 D Y R1 R2 D Z Y∗ W∗ D∗ R∗ 2 D∗ R∗ 1 G∗3 G∗1 . 19
Then after conjugating by wk we have w−1 k µ g X g∗ ¶ wk = D∗ R∗ 2 G∗ 3 G∗1 R∗1 Y∗ W∗ D Z R 1 R2 D∗ W G1 Y G3 D , which is an element in Hwk := w−1 k P wk∩ H. Denote by h∗ = ν(th−1)ν,
where h is as in (3.1). Then diag(h, h∗) ∈ H.
Now the admissibility implies that (3.2) | det g|s2δ
1 2
P(g)τ (g) = ψ(trR2)δH−1wk(h).
Denote by f = diag(D∗, G∗
1, D) and f∗ = diag(D∗, G1, D) such that
diag(f, f∗) is an element in the Levi part of Hwk. Denote by
R = I2n−2k R∗2 G∗ 3 I4k−2n R∗1 Y∗ W∗ I 2n−2k Z R1 R2 I2n−2k W I4k−2n Y G3 I2n−2k an element in the unipotent radical of Hwk. Then we have
f Rf−1 = I ∗ ∗ I ∗ ∗ ∗ I DZDνtν DR 1νG−t1 ν DR2νD−tν I G1W νDtν I DY νDtν DG 3G−11 I .
By Eq. (3.1), we obtain the following conditions: νY + WtJW − Ytν = 0, and G
3 = −νWtJ.
It follows that the dimension of Y is (2n−2k)(2n−2k+1)2 , the dimension of Z is (2n−2k)(2n−2k−1)2 , and the dimensions of R1 and W are (4k − 2n) ×
(2n − 2k). Now it is an easy calculation to show that δ−1 Hwk(h) = | det D|2(2n−2k)+2(4k−2n)= | det g|2k and | det g|s2δ 1 2 P(g) = | det g| s 2| det g| 2n−1 2 = | det g|n+ s−1 2 . Eq. (3.2) becomes | det g|n+s−12 τ (g) = ψ(trR 2)| det g|2k. 20
Hence in order for the double coset P wkH with the condition that
2k ≥ n to be admissible, the irreducible representation τ must has a nonzero θN2n−2k-model when s = 1. This proves Theorem 1.3.
3.2. On the existence and the uniqueness. From the proof of The-orem 1.3 given above, we have the following existence result.
Corollary 3.1. Let τ be an irreducible, unitary, admissible represen-tation of GL2n. Assume that τ has a nonzero θN2n−2k-functional for some k with [n
2] ≤ k ≤ n. Then the induced representation
IndSO4n
P2n (τ ⊗ | det |
s
2)
has nonzero generalized Shalika functionals only when s = 1.
By comparing with Theorem 1.1, we may ask if such generalized Shalika functionals exist on the induced representation
IndSO4n
P2n (τ ⊗ | det | 1 2) and if they are also unique.
For the existence, we may assume that the representation τ has a nonzero θN2n−2k-functional with the smallest possible k satisfying [
n
2] ≤
k ≤ n. Then the corresponding admissible double coset produces a nonzero quasi-invariant functional with s = 1. It is technical to show that such a quasi-invariant functional is in fact supported on a closed P ×H-stable subset of SO4n, which implies that it extends to a nonzero
generalized Shalika functional on the induced representation with s = 1. Hence we will not pursue that matter in this paper. We leave further discussion towards the end of Section 4 related to Theorem 4.2.
Recall that the local uniqueness of the generalized Shalika function-als for irreducible admissible representations of SO4n was proved in
[N10]. However, the representation IndSO4n
P2n (τ ⊗ | det | 1
2) may be re-ducible. From the proof of Theorem 1.3, it is easy to see that such a local uniqueness problem reduces to the following problem on inter-section of different models in the family of θNr-models for irreducible
admissible representations of GL2n. It is not clear to us in general if
the models in this family are disjoint in the sense that there exists no irreducible admissible representation τ having more than one models in this family, except that when τ is supercuspidal. In this case, τ can not have θNr-models for 0 < r < n because of the supercuspidality of
τ . Since an irreducible supercuspidal representation GL2n is generic,
i.e. has a nonzero Whittaker model. By [HR90], the symplectic model, which is the θN0-model here, is disjoint with the Whittaker model for GL2n. Hence an irreducible supercuspidal representation τ of GL2n has
the only possibility to have the θNn-model, which is the Shalika model.
This leads to the uniqueness assertion in Theorem 1.1. In general we can only see that if the representation τ of GL2n admits exactly one of
θNr-model in the family, then the induced representation at s = 1 has
at most one generalized Shalika functional up to scalar.
In order to extend this uniqueness to the case with more general representation τ , we prove a more general disjointedness result (Theo-rem 3.2) below. To state the result, we denote the standard maximal parabolic subgroup of GLm by
Pr,m−r = Mr,m−rNr,m−r
for any integer 0 ≤ r ≤ m with m ∈ N. Also, the Whittaker character ψUm of Um is given by
ψUm(u) = ψ(
m−1X
i=1
ui,i+1), for u = (ui,j) ∈ Um.
For 2k ≤ m, and given any representation (not necessarily irre-ducible) τm−2k of GLm−2k, define
π(τm−2k) = IndGLAm,2km (τm−2k⊗ 1Sp2k ⊗ 1Nm−2k,2k),
where
Am,2k = (GLm−2k⊗ Sp2k)Nm−2k,2k
whose elements are expressed in matrix form as µ
GLm−2k ∗
0 Sp2k
¶ .
Theorem 3.2. For 0 < 2k ≤ m and any representation τm−2k of
GLm−2k, the Whittaker model of GLm and the model π(τm−2k) of GLm
are disjoint in the sense that there is no GLm-intertwining mapping
between the Whittaker model and the model π(τm−2k) of GLm.
Proof. When m = 2k, this theorem reduces to the disjointedness of Whittaker model and Symplectic model, which is known by the result of Klyachko for finite field case and by Heumos and Rallis for p-adic field case ([HR90]).
Now we consider the disjointedness between Whittaker model and π(τm−2k) for 0 < 2k < m. Let
P = Pm−2k,2k = Mm−2k,2kNm−2k,2k.
Let S0 be a complete set of representatives for W (P)\W (GL
m), where
W (G) denotes the Weyl group of a group G. By Bruhat decomposition, P\GLm/Um = ∪w∈S0PwUm,
where Um denotes the upper triangular unipotent subgroup of GLm.
Let
Λ = {diag(Im−2k, B)|B ∈ B0},
where B0 is a complete set of representatives for Sp
2k\GL2k. Then
Am,2k\GLm/Um = ∪w∈S0,λ∈ΛAm,2kwλUm.
For any w ∈ S0 and diag(I
m−2k, B) ∈ Λ, we claim that
Am,2kdiag(Im−2k, B)wUm
is not an admissible coset. Let D0 = µ Q0 R0 T0 ¶ ∈ Am,2k, where Q0 ∈ GL m−2k and T0 ∈ Sp2k. Write D = µ Q R T ¶ = diag(Im−2k, B)−1D0diag(Im−2k, B).
If the conjugate action of w−1 on D (which maps D to w−1Dw)
sepa-rates T into more than two blocks, then some element n in the unipo-tent radical of P will make ψUm(n) 6= 1, and the coset is not admissible.
We use the following example to explain the idea and the method. Let Q = µ Q1 Q2 Q3 Q4 ¶ , T = µ T1 T2 T3 T4 ¶ and R = µ R1 R2 R3 R4 ¶ .
Assume that w = (1, 4), i.e. the permutation of the first and the fourth blocks, then w−1Dw = T4 0 T3 0 R4 Q4 R3 Q3 T2 0 T1 0 R2 Q2 R1 Q1 .
For some element D with nontrivial R3 part in w−1Dw, we have
ψUm(w
−1Dw) 6= 1
and the coset is not admissible.
Next we assume that the conjugate action of w−1 keeps the
symplectic part T block a whole piece (may perform a permutation on the T -block’s interior). Let w0 = w|
Sp2k be the restriction of w to Sp2k-part (as its embedding in GLm) and consider the restriction of this coset
to its corresponding GL2k part (i.e. the coset Sp2kBw0U2k in GL2k).
Then we can see this coset can not be admissible by the disjointedness of Whittaker model and Symplectic model.
This completes the proof. ¤
Now we apply Theorem 3.2 to the case of the mixed model θNr-model
in GL2n. Let m = 2n, and for 0 ≤ 2r < 2n, take
τ = τ2r = IndGLSr2r(ψSr).
It is clear from induction by stages that IndGL2n
Nr (θNr) = Ind
GL2n
(GL2r⊗Sp2n−2r)N2r,2n−2r(τ2r⊗ 1Sp2n−2r⊗ 1N2r,2n−2r). As a consequence of Theorem 3.2, we have the following Proposition. Proposition 3.3. For 0 ≤ r < n, the θNr-model and Whittaker model
are disjoint as representations of GL2n.
Further discussion of relations between the θNr-models and the
Kly-achko models will be given in §5.
Finally the uniqueness for the induced representation in this case follows.
Corollary 3.4. Let τ be an irreducible, unitary, generic representation of GL2n. Then the induced representation
IndSO4n
P2n (τ ⊗ | det |
s
2)
has at most one nonzero generalized Shalika functional up to scalar. If it exists, then s = 1 and τ has a nonzero Shalika functional.
When τ is a non-generic representation of GL2n, such a uniqueness
will be a more technical issue. We omit the discussion here.
We remark that it is also very interesting to study further properties of this new family of models, θNr-models, and find applications to
rep-resentation theory and automorphic forms, following the lines of ideas in [OS07], [OS08-1], and [OS08-2].
4. Conjectures Related to the Generalized Shalika Models
We are going to discuss relations of our previous work on the gen-eralized Shalika models to the local Langlands functoriality and state conjectures on these issues.
The study of the generalized Shalika models for irreducible admissi-ble representations of SO4n over a p-adic local field F can be viewed as
a special case of establishing the general theory of representations and harmonic analysis related to spherical varieties. In this case, the gener-alized Shalika subgroup H2n is a spherical subgroup of SO4n. From the
general theory of spherical varieties ([K96]), from the context of the geometric Langlands program ([GN10] and [GN09]), and from the har-monic analysis on spherical functions ([S08]), for each spherical variety
X attached to a reductive algebraic group G over F , there exists a dual group G∨
X which is a subgroup of the Langlands dual group LG of G,
such that the irreducible admissible representations of G attached to the spherical variety X are conjecturally parametrized in terms of the dual group G∨
X through the local Langlands functoriality conjecture.
In the summer of 2009, in the workshop on Relative Trace Formula and Periods of Automorphic Forms at The American Institute of Math-ematics, Y. Sakellaridis gave a wonderful lecture announcing his joint work with A. Venkatesh on periods and harmonic analysis on spherical varieties, which motivated us to consider the relative functorial parame-trization of distinguished representations of p-adic groups. In a sense, the conjectures and the results discussed here support their more gen-eral conjectures on Plancherel formula on spherical varieties. We are taking a close look at the results obtained in terms of the generalized Shalika models from this perspective.
Recall that the complex dual group of F -split SO4n is SO4n(C). Let
WF be the local Weil group of F . Recall from [A05] that a local Arthur
parameter is a homomorphism
ψ : WF × SL2(C) × SL2(C) → SO4n(C)
such that the restriction of ψ to SL2(C) × SL2(C) is algebraic and
the restriction of ψ to WF × SL2(C) is a tempered local Langlands
parameter. Assume that the local Arthur conjecture holds, that is, there exists a finite set of irreducible admissible representations of SO4n
attached to the local Arthur parameter ψ, which is called the local Arthur packet of ψ and is denoted by Π(ψ). For each local Arthur parameter ψ, one defines a local Langlands parameter
φψ : WF × SL2(C) → SO4n(C) by φψ(w, h) := ψ(w, h, µ |w|12 0 0 |w|−1 2 ¶ ).
For this local Langlands parameter φψ, following the local Langlands
conjecture, there exists a finite set of irreducible admissible represen-tations of SO4n, which is denoted by Π(φψ) and is called the local
L-packet associated to φψ. Conjecturally, the local L-packet Π(φψ)
should be contained in the local Arthur packet Π(ψ). See the recent work of Moeglin ([M09]) for more discussion on this issue.
In general, the local Arthur parameter ψ is a direct sum of stable local Arthur parameters
(4.1) ψ = ψ1¢ ψ2¢ · · · ¢ ψr
where the stable local Arthur parameters ψi (i = 1, 2, · · · , r) are given
by
ψi := (ρi, ai, bi)
where ai and bi are positive integers, which are the dimensions of the
irreducible representations of the two copies of SL2(C), respectively,
and ρi is irreducible continuous homomorphism from the local Weil
group WF to GLdρi(C).
We first consider the result from Theorem 1.1 obtained in [JQ07]. The Langlands quotient LSO4n(1, τ ) with τ an irreducible supercuspidal representation of GL2n has a stable local Arthur parameter
ψ = (ρτ, 1, 2)
where ρτ is the local Langlands parameter attached to τ by the local
Langlands conjecture for GL2n ([HT01] and [H00]). Of course the local
Langlands parameter for the quotient LSO4n(1, τ ) is φψ = ρτ| · | 1 2 ⊕ ρ∨ τ| · |− 1 2.
In this case, the spherical space X := H2n\SO4n has the dual group
G∨
X = Sp2n(C).
Theorem 1.1 shows that if the Langlands quotient LSO4n(1, τ ) has a nonzero generalized Shalika model, i.e. is (H2n, ψH2n)-distinguished if and only if τ has a nonzero Shalika model. By Theorem 1.1, [JNQ10-1], the irreducible supercuspidal representation τ has a nonzero Shalika model if and only if τ is the image under the local Langlands functor-ial transfer from an irreducible generic supercuspidal representation of SO2n+1. This means that the local Langlands parameter
ψ1,2 := (ρτ, 1)
factors through the complex dual group Sp2n(C) of SO2n+1. Since
Sp2n(C) is also the dual group G∨
X associated to the spherical
vari-ety X = H2n\SO4n, the result in Theorem 1.1 proved in [JQ07] can be
expressed by the following diagram:
(4.2)
WF × SL2(C) × SL2(C) −→ SO4n(C)
& %
G∨
X × SL2(C)
where ψ1,2(WF × SL2(C)) is included in G∨X and the restriction to
the second copy of SL2(C) of the stable local Arthur parameter ψ =
(ρτ, 1, 2) which is denoted by ψ3, is the identity homomorphism from
SL2(C) onto SL2(C). This interpretation leads the following conjecture. 26
Conjecture 4.1. Let ψ be a local Arthur parameter of SO4n. Let
Π(φψ) be the local L-packet associated to the local Langlands parameter
φψ, which is included in the local Arthur packet Π(ψ) associated to the
local Arthur parameter ψ. Then there exists a member in Π(φψ) having
a nonzero generalized Shalika model if and only if the local Arthur para-meter ψ factors through G∨
X× SL2(C), which is a subgroup of SO4n(C),
i.e.
ψ(WF × SL2(C) × SL2(C)) ⊂ G∨X × SL2(C).
Furthermore, the projection to the SL2(C) of ψ(1 × 1 × SL2(C)) yields
the identity homomorphism from the second copy of SL2(C) in WF ×
SL2(C) × SL2(C) onto SL2(C).
Next, we interpret the new results obtained in this paper along the line of Conjecture 4.1. We start with Theorem 1.2. If an irreducible admissible representation σ of SO4n has a nonzero generalized Shalika
model, then it can be realized as a quotient of an induced representation of SO4n of type
(4.3) IndSO4n
P2n (τ | det | 1 2)
where τ is an irreducible admissible representation of GL2n. Let φτ
be the local Langlands parameter associated to τ by the local Lang-lands conjecture for GL2n (the Local Langlands Reciprocity Theorem
of Harris-Taylor ([HT01]) and of Henniart ([H00])). Then the expected local Langlands parameter φσ for σ should be
φσ = φτ| · | 1 2 ⊕ φ∨ τ| · |− 1 2.
Assume that the irreducible admissible representation τ of GL2n is
an Arthur representation, i.e. τ has a local Arthur parameter ψτ. Then
we write ψτ as a direct sum of stable ones
ψτ = ⊕rj=1ψj = ⊕rj=1(ρj, aj, lj).
Then the local Arthur parameter of σ should be ψσ = ⊕rj=1(ρj, aj, 2lj).
In other words, Theorem 1.2 shows, under the local Arthur parame-trization conjecture, that if ψ is the local Arthur parameter of σ such that σ ∈ Π(φψ) (the local L-packet included in the local Arthur packet
Π(ψ)) and if σ has a nonzero generalized Shalika model, then in the expression as a formal sum of stable local Arthur parameters
ψ = ⊕rj=1(ρj, aj, bj),
all the integers bj must be even, i.e. bj = 2lj for j = 1, 2, · · · , r. In
this case, σ can be realized as a quotient of the induced representation
(4.3) such that the irreducible admissible representation τ belongs to the local L-packet Π(φψτ), where the local Arthur parameter ψτ of τ
can be expressed as
ψτ = ⊕rj=1ψj = ⊕rj=1(ρj, aj, lj).
This proves the following relation between the generalized Shalika models and the local Arthur parameters, assuming that the local Arthur parametrization conjecture ([A05] and [M09]) holds.
Theorem 4.2. Let ψ = ⊕r
j=1(ρj, aj, bj) be a local Arthur parameter
of SO4n. Let Π(φψ) be the local L-packet associated to the local
Lang-lands parameter φψ, which is included in the local Arthur packet Π(ψ)
associated to the local Arthur parameter ψ. If there exists a member in Π(φψ) having a nonzero generalized Shalika model, then in the local
Arthur parameter ψ, the integers bj’s are all even.
Based on this, we can explain the result of Theorem 1.3 along the line of Conjecture 4.1, which leads the converse of Theorem 4.2. To this end, we state a conjecture for GL2n.
Conjecture 4.3. Let φτ be the local Langlands parameter associated
to the irreducible admissible representation τ of GL2n. Then τ has a
nonzero θNr-model with 0 ≤ r ≤ n if and only if φτ is of symplectic
type, i.e.
φτ(WF × SL2(C)) ⊂ Sp2n(C).
This conjecture is true when τ is supercuspidal, according to Theo-rem 1.1, [JNQ10-1]. Proposition 3.3 shows that when τ is generic, then if τ has a nonzero θNr-model, then r = n and τ has a nonzero Shalika
model. It is expected that τ is of symplectic type. When r = 0, the symplectic models for the Speh representations of GL2n was considered
in [OS07].
Combining Theorem 1.2 with Conjecture 4.3, the result of Theorem 1.3 predicts that if an irreducible admissible representation σ of SO4n
has a local Arthur parameter ψσ and has a nonzero generalized Shalika
model, then ψσ factors through the subgroup G∨X× SL2(C) of SO4n(C).
This explanation of our new results in this paper supports Conjecture 4.1.
At this point, we do not have any more evidence to support our conjectures. However, it is not hard to see that assuming the local Arthur conjecture, the generalized Shalika models plays a role towards the endoscopy structure of the local Arthur packets via the endoscopy transfers. Such an investigation will lead to the converse of Theorem 4.2.
Another application of the generalized Shalika models to the en-doscopy structures of irreducible generic supercuspidal representations of SO2n+1 was discussed in detail in [JNQ10-2]. We omit any further
discussion here.
The whole theory discussed for orthogonal groups ([JQ07], [JNQ08], [JNQ10-1], and [JNQ10-2]) may be extended to other classical groups. We will go back to these topics in our future work.
5. Further relations between the θNr-models and the
Klyachko models
We continue here to discuss relations between the θNr-models and
the Klyachko models, which extend the results in Theorem 3.2 and Proposition 3.3 in §3.2.
Recall that we denote the standard maximal parabolic subgroup of GLs+t corresponding to partition (s, t) by
Ps,t = Ms,tNs,t,
where Ms,t its Levi subgroup and Ns,t its unipotent radical. Also, Bk=
TkUk is the Borel subgroup of GLk with Levi subgroup Tk and Uk
upper-triangular unipotent subgroup.
For any skew symmetric A ∈ GL2m, define
Sp(A) = {g ∈ GL2m|gtAg = A}.
When A = J2m, write Sp2m = Sp(J2m). Let W (GL2m) be the Weyl
group of GL2m, consisting of elements of permutation matrices. Denote
by
A2m= {σλ | λ is diagonal , σ ∈ W (GL2m), σ2 = 1, σλσ−1 = −λ}.
This set has been introduced before. Here we add the subscript to indi-cate the rank of its elements. Given A ∈ A2m, we define an involution
(5.1) ιA: {1, · · · , 2k} 7→ {1, · · · , 2k} by ιA(s) = t if as,t 6= 0.
Notice that ιA is well-defined and ι2A= id.
Recall the Whittaker character of GLn is given by
ψUn(ui,j) = ψ(
n−1
X
i=1
ui,i+1).
Lemma 5.1. Let l ≥ 2 and π be any representation of GL2k−l (not
necessarily irreducible). Then IndGL2k
[Ul⊗GL2k−l]Nl,2k−l(ψUl⊗ π) and Ind
GL2k
Sp2k(1Sp2k) are disjoint. 29
