Learning Spoken Language Representations with Neural Lattice Language ModelingChao-Wei HuangYun-Nung(Vivian) Chen

(1)

NTUMIULAB

Learning Spoken Language Representations with Neural Lattice Language Modeling

Chao-Wei Huang Yun-Nung (Vivian) Chen

National Taiwan University

[email protected] [email protected]

Code available at https://github.com/MiuLab/Lattice-ELMo

(2)

NTUMIULAB • The idea of LM pretraining is adopted on lattices

• We introduce a lattice language modeling objective

• A 2-stage framework is proposed for learning contextualized representations of lattices efficiently

Highlights

(3)

NTUMIULAB

• Intuitive way for SLU: pipelined approach

Task: Spoken Language Understanding

ASR NLU

• ASR errors affects downstream tasks

We can preserve uncertainty using ASR lattices

(4)

NTUMIULAB

Preserve uncertainty using ASR lattices

• Lattices:

directed acyclic graphs which encode several ASR hypotheses

(5)

NTUMIULAB

Preserve uncertainty using ASR lattices

Using lattices helps

LatticeRNN

LM pre-training helps

ELMo

Can we combine them together?

(6)

NTUMIULAB • Use LatticeLSTM to encode nodes of a lattice

• Ask the model to predict the

outgoing transitions(words) given a node’s representation

• When the lattice has only one hypothesis, this reduces to

normal language modeling

Lattice language modeling

(7)

NTUMIULAB

• So now we can pre-train a LatticeELMo!

Lattice language modeling

• However, LatticeLSTM runs prohibitively slow

• Observation: sequential text is actually a lattice with only one hypothesis

=> normal LM pretraining is also lattice LM pretraining

We can do pre-training in two stages!

(8)

NTUMIULAB

LatticeLSTM

LSTM LSTM LSTM

What a day

Linear

a day <EOS>

the, 1.0 0.8

0.2

Linear

0.9 1.0 1.0

0.1

1.0 1.0

the, 1.0

LatticeLSTM Max pooling

classification

Training Target Task Classifier Stage 1: Pre-Training on

Sequential Texts

Stage 2: Pre-Training on Lattices

LatticeLSTM

Two-stage pre-training

(9)

NTUMIULAB ¹⁰⁰ _96.8

72.18

81.48

91.6 91.89

60.54

67.35 94.99

91.98

61.65

68.52

91.69 93.43

61.29

69.95

95.84 95.37

62.88

72.04 95.97

93.29

61.23

67.9

ATIS SNIPS SWDA MRDA

Manual + ELMo 1-best 1-best + ELMo LatticeLSTM Proposed BERT-base

Results

(10)

NTUMIULAB • We extend the sequential LM objective to a lattice language modeling objective

• We propose a 2-stage framework for learning contextualized representations of lattices efficiently

• Experiments on various SLU tasks show that our proposed framework provides consistent improvements

Conclusion

(11)

NTUMIULAB

Thanks for listening!

Code available at https://github.com/MiuLab/Lattice-ELMo

[email protected] [email protected] Chao-Wei Huang Yun-Nung (Vivian) Chen