2014/02/13 Sphinx Lunch

23  Download (0)

Full text


Unsupervised Induction and Filling of Semantic Slot for Spoken Dialogue Systems Using Frame-Semantic Parsing

Yun-Nung (Vivian) Chen, William Yang Wang, and Alexander I. Rudnicky

2014/02/13 Sphinx Lunch

Best Student Paper Award

@ 2013 IEEE Workshop on Automatic Speech Recognition and Understanding – Dec. 9-12, 2013



Introduction Approach Experiments Conclusion



Introduction Approach Experiments Conclusion


In 2011…

“… I have long been looking for good

knowledgeable student to take on the task of getting conversational agents to move beyond prerecorded interaction (or clunky spoken dialog systems) …”

“… Wow! Sounds like they know how to achieve the ultimate machine

intelligence at CMU…”


Spoken Language Understanding (SLU)

 SLU in dialogue systems

SLU maps NL inputs to semantic forms

“I would like to go to Shadyside Tuesday.”

Semantic frames, slots, & values

 often manually defined by domain experts or developers.

location: Shadyside date: Tuesday

What are the problems?


Problems with Predefined Slots

Generalization: may not generalize to real-world users.

Bias propagation: can bias subsequent data collection and annotation.

Maintenance: when new data comes in, developers need to start a new round of annotation to analyze the data and update the grammar.

Efficiency: time consuming, and high costs.

Can we automatically induce semantic slots

only using raw audios?



Introduction Approach Experiments Conclusion


Probabilistic Frame-Semantic Parsing

(Das et al., 2010; 2013) on ASR-transcribed utterances.

First Step



 FrameNet (Baker et al., 1998)

 a linguistically-principled semantic resource, based on the frame-semantics theory.

 Example

Frame (food): contains words referring to items of food.

Frame Element: a descriptor indicates the characteristic of food.

 “low fat milk” 

“milk” evokes the “food” frame;


Frame-Semantic Parsing

 SEMAFOR (Das et al., 2010; 2013)

 a state-of-the-art frame-semantics parser,

trained on manually annotated FrameNet



can i have a cheap restaurant

Frame: capability FT LU: can FE LU: i

Frame: expensiveness FT LU: cheap

Frame: locale by use FT/FE LU: restaurant

The Panacea?

 Unfortunately...

Task: adapting generic frames to task-specific





As A Ranking Problem

 Main idea

 Ranking domain-specific concepts higher than generic semantic concepts

can i have a cheap restaurant

Frame: capability FT LU: can FE LU: i

Frame: expensiveness FT LU: cheap

Frame: locale by use FT/FE LU: restaurant

slot candidate


Slot Ranking Model (1/3)

 Rank the slot candidates by integrating two scores

the frequency of the slot candidate in the SEMAFOR-parsed corpus

the coherence of slot fillers

slots with higher frequency may be more important domain-specific concepts should focus on fewer topics and be similar to each other

lower coherence in topic space higher coherence in topic space

slot: quantity slot: expensiveness

a one

all three


expensive inexpensive


Slot Ranking Model (2/3)

 Measure coherence by word-level context clustering

 For each slot ,

 We have corresponding cluster vector

 Measure coherence measure by pair-wised cosine similarity

slot candidate: expensiveness corresponding value vectors: “cheap”, “not expensive”

(from the utterances with s


in the parsing results)

the frequency of words in v


clustered into cluster k


Slot Ranking Model (3/3)

 Spectral clustering

 For each word

 The approach can be summarized in five steps:

1. Calculate the distance matrix

2. Derive the affinity matrix

3. Generate the graph Laplacian

4. Eigen decomposition of L

5. Perform K-means clustering of eigenvectors



= 1 when w occurs in the i-th utterance r


= 0 otherwise

Reasons why spectral clustering:

1) can be solved efficiently by standard linear algebra

Assume that the words in the same utterance are related to each other



Introduction Approach Experiments Conclusion



 Dataset

 Cambridge University SLU corpus (Henderson, 2012)

 restaurant recommendation in an in-car setting in Cambridge

 WER = 37%

 vocabulary size = 1868

 2,166 dialogues

 15,453 utterances

 dialogue slot (total # = 10): addr, area, food, name, phone,

postcode, price range, signature, task, type


Slot Induction Evaluation

 MAP of the slot ranking model

 measure the quality of induced slots based on induced and reference

slots via the mapping table

Induced Slot Reference Slot

Speak on topic Addr

Part orientational

Area Direction

Locale Part inner outer


Food origin

(NULL) Name

Contacting Phone

Sending Postcode

Commerce scenario

Price range Expensiveness


(NULL) Signature


Task Desiring

Approach MAP

ASR Manual Frequency 67.31 59.41

K-Means 68.45 59.76 Spectral Clustering 69.36 61.86

The majority of the reference slots used in a real

The mapping table between induced and reference slots


Slot Filling Evaluation

 MAP of the slot ranking model

 For each slot, we compute F1by comparing the lists of extracted and reference slot fillers

SEMAFOR Slot Locale by use Speak on topic Expensiveness Origin Direction

Reference Slot Type Addr Price range Food Area

F1-Hard 89.75 88.86 62.05 36.00 29.81

F1-Soft 89.96 88.86 62.35 43.48 29.81

The top-5 F1-measure slot-filling corresponding to matched slot mapping for ASR

F1-Hard: the values of two slot fillers are exactly the same

F1-Soft: the values of two slot fillers both contain at least one overlapping words


Slot Induction and Filling Evaluation

 MAP-F1-Hard / MAP-F1-Soft

 weight the MAP score with F1-Hard / F1-Soft scores

Approach MAP-F1-Hard MAP-F1-Soft ASR Manual ASR Manual Frequency 26.96 27.84 27.29 28.68

K-Means 27.38 27.99 27.67 28.83

Spectral Clustering 30.52 28.40 30.85 29.22

When the induced slot mismatches the reference slot, all the slot

fillers will be judged as incorrect fillers.



Introduction Approach Experiments Conclusion



 We propose an unsupervised approach for automatic induction and filling of semantic slots.

 Our work makes use of a state-of-the-art semantic parser, and adapts the linguistically principled generic FrameNet- style outputs to the target semantic space corresponding to a domain-specific SDS setting.

 Our experiments show that automatically induced

semantic slots align well with the reference slots created

by domain experts.


Thanks for your attention! 

Q & A




Related subjects :