The second stage of the proposed approach is tasked with classifying constituents by a constituency graph. To utilize relevant constituent structures in classifying each constituent, we propose to use a special recursive network, BRNN-CNN, as the model underlying the stage. By following bottom-up links, the model captures the semantic composition of local information for each constituent. By following top-down links, the global information of the structures containing each constituent is captured. Together, the bidirectional passes propagate relevant information on a constituency graph for the identification of named entity constituents.
5.1 Feature Extraction
For every node in a constituency graph, local features are drawn from itself, its left sibling, and its right sibling. Features in use include words, head words,
33
and parse tags. However, features are not always available because of the following reasons.
• Absent siblings,
• Absent words for nonterminal nodes, and
• Absent words and head words for added pyramid nodes.
Should these cases happen, dummy feature values are used.
Besides, total number of tokens in a sentence is used as a global feature.
5.1.1 Word-Level Features
For word-level features, a function N1 is first defined such that it maps each distinct token to a distinct index. Suppose there are n distinct tokens in the corpus.
Then their indices are from 1 to n. A special index 0 is mapped for non-existent tokens.
With the index mapping defined, the word-level features are extracted as the following. For each node i with left sibling j and right sibling k, its word-level features
xi = (N1(wordi), N1(headi), N1(headj), N1(headk))
where wordi denotes the word of i, and headi, headj and headk denote the head words of i, j, and k respectively.
For example, suppose the word-to-index mapping used is shown in Table 5.1.
5.1. FEATURE EXTRACTION 35
For node#5 in Figure 5.1, its word-level features
x5 = (N1(word5), N1(head5), N1(head2), N1(headφ)
= (N1(φ), N1(Kennedy), N1(senator), N1(φ))
= (0, 3, 1, 0)
where φ represents non-existent nodes and tokens.
x φ senator Edward Kennedy Bob and Mary Schindler
N1(x) 0 1 2 3 4 5 6 7
Table 5.1: An example of word-to-index mapping. φ represents a non-existent token.
Figure 5.1: Binarized tree for S1 = (senator, Edward, Kennedy).
5.1.2 Character-Level Features
For character-level features, every word is treated as a sequence of characters, and in the case when a token is non-existent, an empty sequence is used as its character sequence. To facilitate batch computing, words are preprocessed so that they are uniform in length. This is achieved with the aid of a special padding character. In addition, special end and start characters are used to provide boundary information for shift-window models. Algorithm 3 shows the steps of the process. Effectively, for words that are too long, its trailing characters are cut off before prepending start and appending end. Conversely for those short words, start and end are added before appending additional paddings. The uniform length is set to 20, with which the completeness of most dictionary words are preserved and the noisy tails of long tokens such as web addresses are truncated.
Algorithm 3 Unification
1: function UNIFY(word)
2: word ← [start] + word[1..18] + [end]
3: word ← word + [padding] × (20 − word.length)
4: return word
With preprocessed words, a function N2 is defined such that it maps each distinct character to a distinct index. Suppose there are n distinct characters in the corpus.
Then their indices are from 3 to n+2. The indices 0, 1, 2 are reserved for the characters padding, end, and start respectively. For simplicity, the notation is abused so that N2 could also represent a procedure that takes a character sequence c and returns a sequence of mapped indices of U N IF Y (c).
5.1. FEATURE EXTRACTION 37
With the procedure defined, the character-level features are extracted as the following. For each node i with left sibling j and right sibling k, its character-level features
ci = (N2(wordi), N2(headi), N2(headj), N2(headk))
where wordi denotes the word of i, and headi, headj and headk denote the head words of i, j, and k respectively.
For example, suppose characters a, . . . , z, A, . . . , Z are mapped to 3, . . . , 28, 29, . . . , 54 and the uniform length is set to 5 for the purpose of demonstration. Then senator, Kennedy, and non-existent tokens are unified to (start,s,e,n,end), (start,K,e,n,end), and (start,end,padding,padding,padding) respectively. For node#5 in Figure 5.1, its character-level features
c5 = (N2(word5), N2(head5), N2(head2), N2(headφ)
= (N2(φ), N2(Kennedy), N2(senator), N2(φ))
= ((2, 1, 0, 0, 0), (2, 39, 7, 16, 1), (2, 21, 7, 16, 1), (2, 1, 0, 0, 0)) where φ represents non-existent nodes and tokens.
5.1.3 Parse Tag Features
For parse tag features, a function N3 is defined such that it maps each distinct parse tag to a distinct index. Suppose there are n distinct parse tags in the grammar.
Then their indices are from 1 to n. Non-existent parse tags are mapped to index 0.
For each node i with left sibling j and right sibling k, its parse tag features pi = (N3(posi), N3(posj), N3(posk))
where posi, posj, and posk denote the parse tag of i, j, and k respectively.
For example, suppose the pos-to-index mapping used is shown in Table 5.2. For node#5 in Figure 5.1, its parse tag features
p5 = (N3(pos5), N3(pos2), N3(posφ)
= (N3(N P ), N3(N N P ), N3(φ))
= (5, 3, 0)
where φ represents non-existent nodes and parse tags.
p φ NN NNS NNP NNPS NP PYRAMID
N3(p) 0 1 2 3 4 5 6
Table 5.2: An example of pos-to-index mapping. φ represents a non-existent parse tag. Added pyramid nodes all have the same sepcial tag.
5.1.4 Lexicon Hit Features
In addition to the aforementioned features derived from parse trees. Additional lexicon hit features are introduced by external lexicon resources. For each node, there is a feature per lexicon. If the constituent of the node can be found in a lexicon, then the lexicon feature value is set to 1 (hit); 0 (not hit), otherwise. All phrases are lower-cased before deciding equality.
For example, suppose there are three lexicons shown in Table 5.3. Then for
5.2. BRNN-CNN 39
node#5 in Figure 5.1, its lexicon hit features lex5 = (1, 0, 0) because its constituent, Edward Kennedy, is found in the first lexicon.
PER
Table 5.3: Three example lexicons of PER, ORG, and LOC.
5.2 BRNN-CNN
A special recursive neural network, BRNN-CNN, is proposed to classify each constituent from a constituency graph. To classify a node, BRNN-CNN does not only consider its features. Instead, BRNN-CNN considers the hidden states of the node, which are recursively computed from the features of its relevant linguistic structures.
To recursively compute hidden states, constituents must be structured as a di-rected acyclic graph. Then, BRNN-CNN could repeatedly apply its hidden layers from the sources to the sinks of the DAG. For each sentence, two DAGs are formed from its hierarchical constituency graph. One is formed by taking all the nodes and bottom-up links, and the other uses top-down links instead.
Because local information is propagated bottom-up according to constituency structures, bottom-up hidden states capture the semantic composition of each con-stituent. On the other hand, the features of an ancestry and their siblings are propagated down to each descendent, hence top-down hidden states capture global information for each node. These hidden states of each node together contain the structure information of a sentence relevant to the classification of the corresponding constituent.
5.2.1 Input Layer
The extracted features, described in the previous section, are first processed into real-valued vectors by the input layer of BRNN-CNN. For each node, its feature vector is the concatenation of vectors representing word-level, character-level, parse tag, and lexicon hit features.
Word-Level Vector
Suppose there are n distinct tokens in the corpus and the desired word embedding dimension is dx. Then BRNN-CNN stores a word embedding look-up table Wx which is a n-by-dx real-valued matrix. Effectively, every row of Wx represents a word embedding.
Recall that each word-level feature is simply a word index. BRNN-CNN trans-forms each word index into a n-dimensional one-hot vector except for 0. The special index 0 is transformed to a zero vector. Then the vector is multiplied by Wx to retrieve the word embedding. Finally, for a node i with word-level features xi, a
5.2. BRNN-CNN 41
vector Xi is formed by concatenating the embeddings of the 4 words in xi. For example, suppose dx = 2 and the word embedding look-up table
Wx =
Then the word embedding of word index 3 is computed by
It is more complex to compute a character-level vector for a node than other vectors in the input layer. In a nutshell, BRNN-CNN forms a matrix from the character sequence of each token and put it through a series of convolution, max-pooling, and highway layers. These computations actually consist the CNN part of the model, whereas the so-called input layer is actually the input layer of BRNN.
The first step is to form a character-level feature matrix for each token. Recall that for each token, its character-level features is a sequence of character indices.
BRNN-CNN forms a one-hot vector for each character index except for the index 0, which is transformed into a zero vector. These character vectors are then put together into a m-by-n matrix, where m is the uniform word length and n is the number of distinct characters in the corpus plus end and start.
Then sub-word patterns are captured by putting the character-level feature ma-trix through multiple convolution kernels in parallel. Kernels might have different heights as their window sizes, but their widths must be n. For example, if a kernel has height h, the convolutional layer will compute an (m-h+1)-by-1 feature map.
The feature map will then be max-pooled into a scalar, representing the signal strength of a sub-word pattern of length h. Finally, results of all the kernels are put into a vector, called u, of which length, called dc, is the number of kernels.
However, as suggested by Kim et al. [15], the CNN-computed character-level feature vector u of each token is put through an additional highway layer. The final character-level vector of a token, called v, is computed as the following.
t = σ(Wtu + bt)
v0 = ReLU (Wvu + bv)
v = (1 − t) u + t v0
σ(x) represents the sigmoid function 1
1 + e−x, and ReLU (x) = max(0, x). Wt and Wv are dc-by-dcsquare weight matrices, and btand bvare dc-dimensional bias vectors.
Essentially, the final vector v is a weighted sum of its input u and the non-linear transformation v0 of u. By initializing btto a negative value, the layer initially sends its input direct to output like a highway.
In summary, for each node i, BRNN-CNN computes a character-level vector for each of the 4 character index sequences in ci. These 4 vectors are then concatenated to form the character-level vector of the node, called Ci, for the input layer.
5.2. BRNN-CNN 43
Parse Tag Vector
A parse tag vector is computed for each node i. Suppose there are dp distinct parse tags in the corpus. For each parse tag index in pi, a dp-dimensional one-hot vector is formed. The exception is that index 0 is transformed into a zero vector.
These vectors are concatenated into a long vector, called Pi, for the input layer.
Global Word Feature Vector
Aside from a constituent itself and its ancestors, the whole sentence provides useful additional information for NER. BRNN-CNN averages the word-level vectors of all tokens in a sentence as Mx. Similarly, the character-level vectors of all the tokens in a sentence are averaged to get another mean embedding Mc. These two vectors are used for every node in the constituency parse of the sentence as global knowledge.
Input Layer Vector
Finally, for each node i, BRNN-CNN computes its input layer by Equation 5.1.
Ii = XikCikPiklexikMxkMc. (5.1)
The dimension of Ii is given by
dI = dx× 4 + dc× 4 + dp× 3 + dlex+ dx+ dc
where dlex is the number of lexicons.
5.2.2 Hidden Layers
Having computed the input layer for every node on a constituency graph, BRNN-CNN recursively computes two hidden states for every node.
For each node i with the set of child nodes N and the parent p, the bottom-up hidden vector Hbot,i and top-down hidden vector Htop,i are recursively computed by Equation 5.2 and Equation 5.3 respectively.
Hbot,i= ReLU ((IikX
j∈N
Hbot,j)Wbot+ bbot) (5.2)
Htop,i = ReLU ((IikHtop,p)Wtop+ btop) (5.3)
The function ReLU (x) represents max(0, x). Suppose dH is the desired hidden feature dimension. Then Wbot and Wtop are (dI+dH)-by-dH weight matrices. bbot and btop are dH-dimensional bias vectors. If the set of child nodes are empty, a dH-dimensional zero vector is used instead of X
j∈N
Hbot,j. Similarly if i has no parent, a zero vector is used instead of Htop,p.
The computations of the bottom-up and top-down hidden states of node#5 in Figure 5.1 are illustrated in Figure 5.2 and 5.3. The full bottom-up and the top-down passes on the constituency graph in Figure 5.1 are illustrated in Figure 5.4 and Figure 5.5.
5.2. BRNN-CNN 45
Figure 5.2: The bottom-up hidden layer applied to node#5 in Figure 5.1.
Figure 5.3: The top-down hidden layer applied to node#5 in Figure 5.1.
Figure 5.4: The bottom-up hidden layer applied to the graph in Figure 5.1.
Figure 5.5: The top-down hidden layer applied to the graph in Figure 5.1.
5.2. BRNN-CNN 47
Deep Hidden Layers
There can be more than one recursive hidden layer in BRNN-CNN. One-layered recursive neural networks is already deep in the sense of recursion: the hidden layer is stacked as many times as the height of the input hierarchy. Having multiple hid-den layers, however, makes more powerful transformations between two neighboring nodes possible.
Suppose there are 3 hidden layers with desired dimension dH1, dH2, and dH3. For each node i with parent node p, the top-down hidden state features can be computed by the following.
Htop,i,1 = ReLU ((IikHtop,p,1)Wtop,1+ btop,1) Htop,i,2= ReLU ((Htop,i,1kHtop,p,2)Wtop,2+ btop,2) Htop,i,3= ReLU ((Htop,i,2kHtop,p,3)Wtop,3+ btop,3)
ReLU (x) = max(0, x). Wtop,1, Wtop,2, and Wtop,3 are (dI+dH1)-by-dH1, (dH1+dH2 )-by-dH2, and (dH2+dH3)-by-dH3 weight matrices respectively. btop,1, btop,2 and btop,3 are dH1, dH2, and dH3-dimensional bias vectors respectively. Figure 5.6 illustrates the three-layered computation. The bottom-up direction is generalized similarly.
Figure 5.6: The top-down deep hidden layer applied to node i with parent p.
5.2.3 Output Layer
The output layer of BRNN-CNN identifies named entity constituents. For each node, the input bottom-up and top-down hidden state vectors contain relevant local and global information of the corresponding constituent. The output is the predicted probability distribution of named entity classes.
Given the set of named entity categories C with size n, a function N4 is first defined to map each distinct NE category to a distinct integer between 1 . . . n. The number n + 1 is reserved for the special category NON NE. The inverse function mapping integers to categories is denoted by N4−1. Therefore the dimension of predicted distributions dO= n + 1.
For any node x, let Hx = Hbot,x+ Htop,x. And let σ denote the softmax function where, for any vector x, σ(x)t = ext
X
u
exu. Then for each node i with left sibling j and right sibling k, its class probability distribution is given by Equation 5.4.
Oi = σ((HikHjkHk)Wout+ bout) (5.4)
Wout is a (dH × 3)-by-dO weight matrix. bout is a dO-dimensional bias vector. If a sibling does not exist, zero vectors are used as its hidden states. Should deep hidden layers be deployed, the last hidden layer is used. Figure 5.7 illustrates the computation of the output layer.
5.3. PREDICTION COLLECTION 49
Figure 5.7: The output layer applied to node i with left sibling j and right sibling k.
The fully-connected layer does not contain non-linear transformations like ReLU.
5.3 Prediction Collection
After BRNN-CNN produces a probability distribution for each constituent, the set of predicted named entities are now collected from the constituency graph.
For each constituent, the system classifies it as the category with highest pre-dicted probability. Formally, for each node i, the system predicts its label by Equa-tion 5.5.
Li = N4−1(argmax
j
Oij) (5.5)
Oij represents the j-th element of Oi.
Intuitively, the predictions of a sentence are the label Li for every node i unless Li = N ON N E. Given the set of sentences S and the set of named entity categories
C, the set of predicted named entities
Ea= {(i, (j, k), Ln) | Si ∈ S, n ∈ GNSi, (Sij, . . . , Si(k−1)) = constn, Ln∈ C}
where GNSi denotes the set of nodes in the constituency graph of the sentence Si, and constn denotes the corresponding constituent of the node n.
In many applications, overlapping named entities, including nested named enti-ties, are not practically useful and should not be predicted by NER systems. These could happen to the above NE collection method. For instance, two NEs are nested if their corresponding nodes are an ancestor and its descendant.
To form non-overlapping predictions, a special NE collection scheme is applied by the system. The scheme traverses through every node of a constituency graph in two passes. The first pass performs a depth-first walk from the root node of the original constituency parse tree in the graph. The system stops recurring down a branch as soon as it encounters a node i in the branch of which Li 6= N ON N E. Then a second pass walk through the additional pyramid nodes in a breadth-first fashion, collecting an additional NE only if it does not overlap with previously collected NEs.
Briefly speaking, the system forms non-overlapping predictions from the output of BRNN-CNN by preferring tree nodes over additional pyramid nodes and larger NEs over smaller ones. The heuristics behind this scheme is to appreciate parsing and to avoid nested NEs, like the first name and the last name of a full name.
Chapter 6
Evaluation
The constituency-oriented approach is evaluated on CoNLL 2003 NER and OntoNotes 5.0 NER. The detailed setup of the experiments is given in the first section. Major results against state-of-the-art systems and related work are shown in the second section. Finally, we analyze different aspects of the approach with ablation studies and discuss their abstractive meaning with case studies.
6.1 Experimental Setup
6.1.1 Parameter Initialization
Parameters of BRNN-CNN are contained in the CNN, the highway layer, the BRNN hidden layers, and the output layer. In addition, the word embedding look-up table is also trainable. Table 6.1 summarizes the weights and biases of which values need to be decided.
51
Except for Wx, the parameters of all other layers are initialized with Xavier initializer [11]. The initializer tries to ensure the scale of output values for deep networks. This is desirable because BRNN-CNN might go down an indefinitely deep recursion.
On the other hand, the initialization of the word embedding look-up table has two cases. First, a pre-trained table is attempted to be used. For example, unsupervised word embeddings trained by GloVe [23] from a 840 billion-token web corpus are available. If a pre-trained GloVe vector could be found for a word, the corresponding row of Wx is initialized by that vector. Otherwise, the Gaussian distribution with zero mean and 0.1 standard deviation is used for sampling.
Layer Weights and Biases
Word Embedding Look-Up Table Wx
CNN Kernels
Highway Layer Wt, Wv, bt, bv Hidden Layers Wbot, Wtop, bbot, btop
Output Layer Wout, bout
Table 6.1: Parameters of BRNN-CNN.
6.1.2 Parameter Optimization
Once initialized, the model parameters can be optimized with a training corpus.
In a training corpus, all ground truth named entities are known to BRNN-CNN.
6.1. EXPERIMENTAL SETUP 53
For each node n in the constituency graph of a sentence, its ground truth NE label is denoted by Ln,g. If n corresponds to a named entity e = (i, (j, k), c), Ln,g = c.
Otherwise, Ln,g = N ON N E. Equation 6.1 gives the loss function of a node.
lossn= − log OnN4(Ln,g) (6.1)
OnN4(Ln,g) denotes the N4(Ln,g)-th element of On. The objective is set to minimize the average loss of all nodes for a corpus. To achieve this, Adam optimizer [16] is used for parameter updates.
To avoid overfitting for gradient descent optimization algorithms such as Adam, it is desirable to stop iterating parameter updates with the aid of a validation corpus.
Let F denotes an evaluation function, Ea denotes the prediction of BRNN-CNN for the validation corpus, and Eg denotes the set of ground truth NEs of the corpus.
After each round, or epoch, of parameter update with the training corpus, the validation score F (Ea; Eg) is checked. Since initialization, the best score is kept. If a record has not been broken for 20 epochs, the training stops and the parameter values that achieved the best score are restored.
Dropout
Zeroing some of the output of some network layers is oftentimes beneficial to training. BRNN-CNN is no exception and dropout layers are added for the input
Zeroing some of the output of some network layers is oftentimes beneficial to training. BRNN-CNN is no exception and dropout layers are added for the input