Considering the benefits brought by MF techniques, including 1) modeling noisy data, 2) modeling hidden semantics, and 3) modeling long-range dependencies between observations, in this work we apply an MF approach to SLU modeling for SDS. In our model, we use U to denote a set of input utterances, W as a set of word patterns, and S as a set of semantic slots that we would like to predict. The pair of an utterance u ∈ U and a word pattern/semantic slot x ∈ {W ∪ S}, hu, xi, is a fact. The input to our model is a set of observed facts O, and the observed facts for a given utterance is denoted by {hu, xi ∈ O}. The goal of our model is to estimate, for a given utterance u and a given word pattern/semantic slot x, the probability, p(Mu,x= 1), where Mu,x is a binary random variable that is true if and only if x is the word pattern/domain-specific semantic slot in the utterance u. We introduce a series of exponential family models that estimate the probability using a natural parameter θu,x and
the logistic sigmoid function:
p(Mu,x= 1 | θu,x) = σ(θu,x) = 1
1 + exp (−θu,x) (6.1)
We construct a matrix M|U |×(|W |+|S|) as observed facts for MF by integrating a feature model and a knowledge graph propagation model below.
6.3.1 Feature Model
First, we build a word pattern matrix Fw with binary values based on observations, where each row represents an utterance and each column refers to an observed unigram. In other words, Fw carries basic word vectors for utterances, which is illustrated as the left part of the matrix in Figure 6.1b.
To induce the semantic elements, we parse all ASR-decoded utterances in our corpus us-ing SEMAFOR [48, 49], and extract all frames from semantic parsus-ing results as slot can-didates [31, 55]. Figure 3.2 shows an ASR-decoded utterance example “can i have a cheap restaurant ” parsed by SEMAFOR. Three FrameNet-defined frames capability, expensiveness, and locale by use are generated for the utterance, which we consider as slot candidates for a domain-specific dialogue system [4]. Then we build a slot matrix Fswith binary values based on the induced slots, and the matrix also denotes slot features for all utterances (right part of the matrix in Figure 6.1b).
To build a feature model MF, we concatenate two matrices:
MF = [ Fw Fs ], (6.2)
which is the upper part of the matrix in Figure 6.1b for training utterances. Note that we do not use any annotations, so all slot candidates are included.
6.3.2 Knowledge Graph Propagation Model
As mentioned in Chapter 3, because SEMAFOR was trained on FrameNet annotation, which has a more generic frame-semantic context, not all the frames from the parsing results can be used as the actual slots in the domain-specific dialogue systems. For instance, we see that the frames expensiveness and locale by use are essentially key slots for the purpose of understanding in the restaurant query domain, whereas the capability frame does not convey particularly valuable information for SLU.
Assuming that domain-specific concepts are usually related to each other, considering global
relations between semantic slots induces a more coherent slot set. It is shown that the relations on knowledge graphs help make decisions on domain-specific slots [39]. Similar to Chapter 4, we consider two directed graphs, semantic and lexical knowledge graphs, where each node in the semantic knowledge graph is a slot candidate si generated by the frame-semantic parser, and each node in the lexical knowledge graph is a word wj.
• Slot-based semantic knowledge graph is built as Gs= hVs, Essi, where Vs= {si ∈ S} and Ess= {eij | si, sj ∈ Vs}.
• Word-based lexical knowledge graph is built as Gw = hVw, Ewwi, where Vw = {wi ∈ W } and Eww= {eij | wi, wj ∈ Vw}.
The edges connect two nodes in the graphs if there is a typed dependency between them. The structured graph helps define a coherent slot set. To model the relations between words/slots based on the knowledge graphs, we define two relation models below.
• Semantic Relation
To model word semantic relations, we compute a matrix RSw = [Sim(wi, wj)]|W |×|W |, where Sim(wi, wj) is the cosine similarity between the dependency embeddings of word patterns wi and wj after normalization. For slot semantic relations, we compute RSs = [Sim(si, sj)]|S|×|S| similarly1. The matrices RSw and RSs model not only semantic similarity but functional similarity since we use dependency-based embeddings [108].
• Dependency Relation
Assuming that important semantic slots are usually mutually related to each other, that is, connected by syntactic dependencies, our automatically derived knowledge graphs are able to help model the dependency relations. For word dependency relations, we compute a matrix RDw = [ˆr(wi, wj)]|W |×|W |, where ˆr(wi, wj) measures a dependency relation between two word patterns wiand wjbased on the word-based lexical knowledge graph, and the detail is described in Section 4.3.1. For slot dependency relations, we similarly compute RDs = [ˆr(si, sj)]|S|×|S| based on the slot-based semantic knowledge graph.
With the built word relation models (RSw and RDw) and slot relation models (RSs and RDs), we combine them as a knowledge graph propagation matrix MR2.
MR=
h RSDw 0 0 RSDs
i
, (6.3)
1For each column in RSwand RSs, we only keep top 10 highest values, which correspond the top 10 seman-tically similar nodes.
2The values in the diagonal of MR are 0 to model the propagation from other entries.
where RSDw = RSw+ RwD and RSDs = RSs+ RDs to integrate semantic and dependency relations.
The goal of this matrix is to propagate scores between nodes according to different types of relations in the knowledge graphs [26].
6.3.3 Integrated Model
With a feature model MF and a knowledge graph propagation model MR, we integrate them into a single matrix.
M = MF · (αI + βMR)
= [ Fw Fs ] ·h αI + βRw 0 0 αI + βRs
i
=
h αFw+ βFwRw 0 0 αFs+ βFsRs
i ,
(6.4)
where M is the final matrix and I is an identity matrix. α and β are weights for balancing original values and propagated values, where α + β = 1. The matrix M is similar to MF, but some weights are enhanced through the knowledge graph propagation model, MR. The word relations are built by FwRw, which is the matrix with internal weight propagation on the lexical knowledge graph (the blue arrow in Figure 6.1b). Similarly, FsRs models the slot correlations, and can be treated as the matrix with internal weight propagation on the semantic knowledge graph (the pink arrow in Figure 6.1b). The propagation models can be treated as running a random walk algorithm on the graphs.
Fs contains all slot candidates generated by SEMAFOR, which may include some generic slots (such as capability), so the original feature model cannot differentiate domain-specific and generic concepts. By integrating with Rs, the semantic and dependency relations can be propagated via edges in the knowledge graph, and domain-specific concepts may have higher weights based on the assumption that slots for dialogue systems are often mutually related [39]. Hence, the structure information can be automatically involved in the matrix.
Also, the word relation model brings the same function, but on the word level. In conclusion, for each utterance, the integrated model not only predicts probabilities that semantic slots occur but also considers whether the slots are domain-specific. The following sections describe the learning process.
6.3.4 Parameter Estimation
The proposed model is parameterized through weights and latent component vectors, where the parameters are estimated by maximizing the log likelihood of observed data [46].
θ∗ = arg max
θ
Y
u∈U
p(θ | Mu)
= arg max
θ
Y
u∈U
p(Mu | θ)p(θ)
= arg max
θ
X
u∈U
ln p(Mu| θ) − λθ,
(6.5)
where Mu is the vector corresponding to the utterance u, because we assume that each utterance is independent of others.
To avoid treating unobserved facts as designed negative facts and to complete missing entries of the matrix, our model can be factorized by a matrix completion technique with a low-rank latent semantics assumption [97, 134]. Bayesian personalized ranking (BPR) is an optimiza-tion criterion that learns from implicit feedback for MF by a matrix compleoptimiza-tion technique, which uses a variant of the ranking: giving observed true facts higher scores than unobserved (true or false) facts to factorize the given matrix [134]. BPR was shown to be useful in learning implicit relations for improving semantic parsing [38, 135].
6.3.4.1 Objective Function
To estimate the parameters in (6.5), we create a dataset of ranked pairs from M in (6.4):
for each utterance u and each observed fact f+ = hu, x+i, where Mu,x ≥ δ, we choose each word pattern/slot x− such that f− = hu, x−i, where Mu,x < δ, which refers to the word pattern/slot we have not observed to be in utterance u. That is, we construct the observed data O from M . Then for each pair of facts f+ and f−, we want to model p(f+) > p(f−) and hence θf+ > θf− according to (6.1). BPR maximizes the summation of each ranked pair, where the objective is
X
u∈U
ln p(Mu | θ) = X
f+∈O
X
f−6∈O
ln σ(θf+ − θf−). (6.6)
The BPR objective is an approximation to the per utterance AUC (area under the ROC curve), which directly correlates to what we want to achieve – well-ranked semantic slots per utterance.
6.3.4.2 Optimization
To maximize the objective in (6.6), we employ a stochastic gradient descent (SGD) algo-rithm [134]. For each randomly sampled observed fact hu, x+i, we sample an unobserved fact hu, x−i, which results in |O| fact pairs hf−, f+i. For each pair, we perform an SGD update using the gradient of the corresponding objective function for MF [68].