Result of the Event Extraction Using Graph-Based Feature Sets

4. Event Extraction Using Graph-Based Feature Sets

4.6 Experimental Result and Discussion

4.6.2 Result of the Event Extraction Using Graph-Based Feature Sets

We submitted our results in the .a2 file format for evaluation. Our results consist of two parts. First, we check the results of our two components of our system independently. Second, we check the results of the extraction systems of the two components combined.

The experimental results from the first major components are shown in Tables 8.

In the table, column “Substitutions” (incorrect slots) is the number of slots in our system that are aligned to slots in the reference and are scored as incorrect. Column

“Deletions” (missing slots or false rejections) is the number of slots in the reference that are not aligned to any slot in our system. Column “Insertions” (spurious slots or false acceptances) represents the number of slots in our system that are not aligned to any slot in the reference. Column “Recall” is the number of correctly aligned slots divided by the number of slots. Column “Precision” is the number of correctly aligned slots divided by the number of all returned aligned slots. Column “F-score” considers both precision rate and recall rate. Column “SER” is the score computed with Equation (5).

Table 8. Result from event extraction using graph-based feature sets

Substitutions Deletions Insertions Recall Precision F-score SER

8 62 9 0.2045 0.5143 0.2927 0.8977

5 Semantic Annotation Based on Natural Language Processing

The Result of the Event Extraction Using Graph-Based Feature Sets represented in Section 4 shows nearly about 90 percent SER score. In an attempt to improve this result to get lower SER score, it is observe that the method of “the Event Extraction Using Graph-Based Feature Sets” uses less semantic features and more syntactic features. Hence, the study considers using the method of “Semantic Annotation Based on Natural Language Processing

.”

Biological sentences are often complex and hard to extract information or knowledge embedded deeply within the sentences without proper inference. In the Semantic Annotation Based on Natural Language Processing approach, three layers of event extraction workflows are used: syntactic analysis, semantic analysis, and inference. The syntactic analysis focuses on extracting the structure in the sentence using the dependency tree from our sentence parsing. The training data is used for semantic analysis and inference.

5.1 Parsing and Preprocessing

Biological events in written literatures are often in a much more complex sentence structures. Extracting events and relations from the complex structures is a difficult task of requiring proper inference. To deal with this complex issue, Stanford dependency parser is used to derive the dependency structure of terms in the input sentence. For the purpose of semantic analysis, semantic tags are assigned to proper nouns of our subjects in a sentence (possible arguments) or to an object. Finally, to

access semantic meanings in a sentence, semantic annotation is used to derive semantic roles of the subject and the object.

5.2 Semantic Annotation

Four different thesauri to assign semantic tags to noun phrases of subjects and objects as described in the work of Huang et al. (2006). The thesauri used are MeSH, Gene Ontology (GO), WordNet and a stopword list.⁸ We use the following semantic codes shown in Table 9.

Table 9. Semantic codes for annotations Semantic Code Description

<ME.D> Descriptors in MeSH

<ME.C> Supplementary Concepts in MeSH

<ME.Q> Qualifiers in MeSH

<SW.A> Articles in Stopword list

<SW.P> Preposition in Stopword list

<SW.C> Conjunction in Stopword list

<SW.N> Noun in Stopword list

<WN.A> Adjectives in WordNet

<WN.AD> Adverb in WordNet

<WN.N> Noun in WordNet

<WN.V> Verb in WordNet

<GO.FUN> Molecular Function in GO

<GO.PRO> Biological Process in GO

<GO.COM> Cellular Component in GO

8 http://jmlr.org/papers/volume5/lewis04a/a11-smart-stop-list/english.stop

The semantic annotation process gives the following result for a particular given sentence in the test data, as shown in Table 10.

Table 10. An example of semantic annotation Input sentence (from PMID-10075739-S9):

Syntactic analysis on the sentences can help provide the clues for subsequent semantic interpretation through their syntactic structures. Stanford full parser was used to parse sentences and generate their dependency tree structures. Referring to the study of Huang et al. (2006), several rules were applied to extract noun phrases, the real verb of

25 auxiliary verb, and such a verb node is selected as our real verb. In other instances, a root verb might be an intransitive verb and may have a clause of a sentence following it. The clause is traversed and searched for other verbs that are present.

To further check if the real verb is a possible event trigger, we identify the subject and the object in the sentence. If a subject and an object relate to the same verb and if the subject or the object of the verb is a named entity, we directly classify our verb in the ontology.

In Figure 5, we get the root verb “Appears” first. When traversing the parse tree, the verb “protect” is a child node of “appears” and has an auxiliary verb “to” among its children. Hence, “protect” should be the real action.

In some situations, we have to deal with compound sentences that have the clauses connected by conjunctions, and the relationships between these clauses are determined by the conjunction. We assume that dependent clauses connected by dependent markers such as “after”, “although”, “as”, “because”, “if”, “since”, “until”,

“when”, “whether”, and “while” may have causal relationship between them. Clauses joined by the coordinating conjunctions such as “and”, “or”, “but”, “so”, and “yet” or joined by independent markers such as “also”, “consequently”, “however”,

“moreover”, and “therefore” are assumed to be independent clauses. These sentences were separated into two clauses and the dependency tree is traversed for extracting the real verb. This real verb presents our identified trigger and we can query our ontology to assign it to a category.

5.4 Semantic Analysis

In this phase, a dictionary built by grouping annotated triggers and filtering out non-trigger tokens. All the given annotated triggers in the training and development datasets were extracted and categorized based on their event types. For each trigger, the confidence score was calculated by counting the frequency it has being annotated as a trigger and its frequency being found in the training dataset. To decide from our trigger candidates, the confidence score of our SVM classifier was calculated using the distance from the hyperline. We use n classes versus all other class approach by having

object

Figure 5. Dependency tree of a sentence “The expression of TRAF2 appears to protect cells from serum deprivation-induced death.”

n classes as a positive class and the rest as a negative class, where n ranges from 1 to the total number of event types. We calculate the average distance from the hyperline for each classification to represent our SVM confidence for classification.

A list of prepositions and adjectives derived from the training and development data via dependency parsing forms our list of non-trigger words from the training dataset. Triggers that belong to the non-trigger list as well as those that consist of more than two words are filtered out. To extract the biological events from the parse tree after obtaining a list of candidate triggers, we adapt two syntactic patterns based on similar concepts as found in extraction of PPIs (Protein-Protein Interactions). The study use the pattern for triggers in noun and verb forms.

5.4.1 Triggers in a Noun Form

In extracting events from a noun phrase (NP), we start with checking if the candidate trigger is a noun. Then we find an NP, which is a joined node of this trigger and at least one named entity from the parse tree. The study focuses on two cases: For the first case, we consider a case where the NP is follow by a prepositional phrase (PP) tag. The second case is considered when the trigger is the head of this NP. Depending on the trigger type (simple, binding or regulatory event), we extract candidate events using the following two basic rules, called Form 1 and Form 2.

Form 1:

where NP means a noun phrase, NN means a noun, PP stands for a prepositional phrase, DT is a determiner, CC means a conjunction, and IN is a preposition. In Form 1 rule, a noun phrase is composed of two nouns. Form 2 rule says that a noun phrase is composed of a noun phrase and a prepositional phrase. The illustration examples of the basic rules are pictured in Figures 6 and 7.

Figure 6. NP pattern example: Form 1

Figure 7. NP pattern example: Form 2 NP

NN NN

PRO1 expression

DT NN

the interaction

PRO0

and PRO1

NN CC between

In Figures 6 and 7, PRO0 and PRO1 stand for some named entities.

For the simple or regulatory event type, if the NP does not have a PP tag, we extract all proteins on the left of the trigger from the NP and form event pairs <trigger, argument>. Otherwise, we extract all proteins on the right of the trigger from the NP and form event pairs <trigger, argument>. For example, <expression, PRO1> is an extraction from Form 1 in Figure 6. For the binding event type, if the NP does not have a PP tag, and if proteins are in a compound form such as PRO0/PRO1, PRO0-PRO1, where PRO0 and PRO1 are entities, then form an event triple <trigger, argument1, argument2>. Otherwise, form events pairs <trigger, argument>. If an NP has a PP tag, which is one of the following preposition pairs, between/and, of/with, of/to, then split the proteins from NP into two lists based on the PP or Coordinating Conjunctions (CC). Form triples <trigger, argument1, argument2> where argument1 and argument2 are from different lists. For example, <interaction, PRO0, PRO1>, as shown in From 2 of Figure 7.

5.4.2 Triggers in a Verb Form

To extract events from a verb phrase (VP), the trigger is checked if it is a verb. If it is, then the next step is to find a VP, which is a direct parent of this trigger from the parse tree, and to find a sister NP immediately preceding this VP. In the following, we extract the candidate events as in the NP case mentioned above.

5.5 Experimental Results and Discussion

5.5.1 Result of semantic annotation based on natural language processing

As in our Event Extraction Using Graph-Based Feature Sets approach, the submitted format is in the .a2 file format for evaluation. The experimental result is shown in Table 11. From Tables 8 and 11, it is observable that the score of Event Extraction Using Graph-Based Feature Sets approach is slightly different from the score of Semantic Annotation Based on Natural Language Processing

.

It is worthto notice that, the Event Extraction Using Graph-Based Feature Sets approach has better SER score because its total slot error is lower. Here, the slot error total is the sum of substitution, deletion and insertions scores.

Table 11. Result of semantic annotation based on NLP

Substitution Deletions Insertions Recall Precisions F-score SER

9 61 10 0.2045 0.4865 0.288 0.9091

In our stacking and filtering component of our system, we concentrate on the important event extraction in the GRN-shared task. Event extraction is a primary objective in the formation of a regulation network. Every biological molecule or genetic entity is associated to a molecular or a genic event. Hence, it is essential to be able to extract biological events and their arguments from any given input sentence.

In our attempt to improve the precision rate and lower the slot error of the system, we combine the output results of event extraction from the two major components of our system. For a given sentence, we check into the output result on the extracted events and their arguments for similarities and differences. For an event with similar

arguments that appear from the two components, the approach retains a single case. In the case where the similar event extracted from the two components but which has different arguments, we keep both cases of the events. The stacking and filtering component functions to combine the two results of extracted events with their respective arguments and removing redundancies. The evaluation score of this result using the shared task’s test evaluation service is given in Table 12 below.

Table 12. System score from the shared task evaluation service with the combined strategy

Substitution Deletion Insertions Recall Precision F-score SER

9 59 10 0.2273 0.5128 0.3150 0.8864

From Table 12, it can be understood that our combined result is dominate by the approach of event extraction using the graph-based feature sets. However, the result of individual components of our system is very much similar. Our semantic annotation approach has a poor score in the number of deletions causing a higher slot error rate.

6 Statistical Approach for Biological Event Extraction Using Markov’s Method

From the above three results: result of Event Extraction Using Graph-Based Feature Sets approach, result of Semantic Annotation Based on Natural Language Processing approach, and the combined result, it is worth to notice the slight differences among the three results. The best score is obtained from the combined result of the two approaches. However, the difference in SER score is not significant enough. In an attempt to a get better result, it is observable the above two approaches in Section 4 and Section 5 do not focus on statistical analysis in combinational linguistics. In an attempt to get a better result, the study uses the statistical analysis using Markov’s Logic.

Markov Logic (Richardson and Domingos, 2006) is a statistical relational learning language based on First Order Logic and Markov Networks. A formalism that extends First Order Logic to allow formulae that can be violated with some penalty.

From an alternative point of view, an expressive template language uses First Order Logic formulae to instantiate Markov Networks for repetitive structures.

We can introduce Markov Logic by considering the event extraction task (as relational structure over token). In Markov Logic, we can model this task by introducing logical predicates such as event(event_token, type) and argu(role, role_filer). We achieve this by using profile Hidden Markov’s Model (profile HMM).

Profile HMM has been well adopted for sequence alignment in computational biology.

However, instead of aiming to identify highly conserved region/motifs of multiple alignments, our profile analysis aims at extracting patterns of occurrence of events and their arguments in training and development data and uses such information as

background observation. Then we specify a set of weighted First Order formulae that define a distribution over sets of the ground atoms of these predicates (so-called possible worlds). Note that we will refer to word token as “observed ” because they are already known in advance. In contrast, annotation and annotation types are as “hidden”

because we also need to infer its ground atoms at test time.

Figure 8 shows the flowchart of the statistical approach for biological event extraction using Markov’s method. The details are in the following subsections.

Figure 8. Statistical Approach for Biological Event Extraction Using Markov’s Method

Input Data

Background Data

Markov Logic Network

Logical Formulae Extractor

Output Preprocessing

6.1 Preprocessing

In the preprocessing level, we take into consideration the given annotation of the training data and the development data. The first approach in the preprocessing is the extraction of the annotations in the training and development data in logical predicate format. The objective of this is to represent event structures as relational structures over tokens. For each input test file, we record the three top text documents in the training and development data that have a close similarity with the input in the test file.

In the next approach of preprocessing, the objective is to obtain any term dependency in the input text data from the shared task’s test files. This is achieved by parsing the input sentences of the test data using Stanford parser. This helps directly deal with word tokens instead of character offset or a sentence token and be able to trace dependency relationship between tokens of interest. It further enables to formulate the probabilistic model in terms of consistent word tokens.

6.2 Markov’s Logic Network for Event Prediction

For each test file given in the shared task, a link of a network is developed to connect it to all the three nearest similar documents in our background data using cosine similarity. This aims at assigning probabilistic weights to the possible event, event triggers, and event argument in the test file. We use a case prediction of entities found in the test data within the dataset created in the preprocessing level. The proposed approach defines a predicate entity(i) to represent an entity token i in the background data. For each entity in the test data, if it is found in the dataset, it is assigned a value of one (true), otherwise assigned zero (false). For any entity with value one assignment, a query for all its associated annotations such as events, relations and all other

associated information that is a target of the shared task in our background data using predicates is excuted. These annotations are assigned a probabilistic value that describes the probability of its appearance in relation to a particular entity within annotations found in our dataset.

From the created dataset, a set of logical predicates such as eventType(event_token, type), and fun(argu(), role) are defined. These predicates assign the weighted scores depending on the appearance of the arguments within the predicates in the processed test input data. The study converts the task into the link prediction over a sequence of predicates describing tokens.

For simplicity, we introduce a formal representation of the events in a sentence and identify the token position in the sentence. We can describe an event e as a tuple (i, t, fun()). Here i is the token position of the trigger word, t is the event type of e and fun() is a set of label arguments (a, r) where each a is either an entity, or event, and r is role a play in with respect to e. For the shared task, five primary annotations are given:

text annotation, event annotation, event modification annotation, relation annotation and normalization annotation. The text annotation defines an entity or event trigger in a specific span of texts and assigns it a type. Event annotation gives annotations of events and relations. Event annotations are a primary extraction target in the main tasks of BioNLP-2013 shared task. Gold standard event annotations are given in both training and development data. Participants are required to create both the event trigger and event annotation for the test data. Event modification annotations contain additional information about other annotations, such as events that are stated speculatively or in a negative context. Relation annotations are similar to event annotations but defined by arguments and types. The normalization annotation gives an external reference to other annotations. In the test data, only the entity and the text

files containing the abstracts are given. For each sentence, we also introduce a state tuple, that represents the sentence as an event structure for a particular sentence represented using a predicate. An example is shown as below:

start (event, eventType, eventTrigger, eventArgument)

For our test data, we make efforts to effectively use both the given text files and the annotation files. Entities in the annotations are assigned binary values during the

在文檔中 Event Extraction for Gene Regulation Network Using Statistical and Semantic Approaches (頁 28-0)