Wide coverage biomedical event extraction using multiple partially overlapping corpora
© Miwa et al.; licensee BioMed Central Ltd. 2013
Received: 11 December 2012
Accepted: 24 May 2013
Published: 3 June 2013
Biomedical events are key to understanding physiological processes and disease, and wide coverage extraction is required for comprehensive automatic analysis of statements describing biomedical systems in the literature. In turn, the training and evaluation of extraction methods requires manually annotated corpora. However, as manual annotation is time-consuming and expensive, any single event-annotated corpus can only cover a limited number of semantic types. Although combined use of several such corpora could potentially allow an extraction system to achieve broad semantic coverage, there has been little research into learning from multiple corpora with partially overlapping semantic annotation scopes.
We propose a method for learning from multiple corpora with partial semantic annotation overlap, and implement this method to improve our existing event extraction system, EventMine. An evaluation using seven event annotated corpora, including 65 event types in total, shows that learning from overlapping corpora can produce a single, corpus-independent, wide coverage extraction system that outperforms systems trained on single corpora and exceeds previously reported results on two established event extraction tasks from the BioNLP Shared Task 2011.
The proposed method allows the training of a wide-coverage, state-of-the-art event extraction system from multiple corpora with partial semantic annotation overlap. The resulting single model makes broad-coverage extraction straightforward in practice by removing the need to either select a subset of compatible corpora or semantic types, or to merge results from several models trained on different individual corpora. Multi-corpus learning also allows annotation efforts to focus on covering additional semantic types, rather than aiming for exhaustive coverage in any single annotation effort, or extending the coverage of semantic types annotated in existing corpora.
Manually created annotations are required as training material for state-of-the-art statistical systems. Manual annotation is time-consuming and expensive, and annotation efforts become increasingly demanding as more types of entities, relations and events are included in the scope of annotation. It is infeasible to deal with all potentially interesting semantic types in any single annotation effort due to cost and time restrictions and the difficulty of maintaining annotation consistency and quality while taking large numbers of semantic types into consideration. Each annotation effort thus tends to focus on a limited number of semantic types relevant to its immediate aims, which in turn results in the proliferation of corpora that overlap only partially in semantic scope, if at all [3-7].
Such partially overlapping semantic annotations represent significant challenges for statistical extraction systems. The combination of corpora whose annotation scopes are different causes difficulties in training, even in cases where semantic types shared between the corpora have been annotated using the same criteria .
Due to these challenges, each system is typically trained on a single corpus that is fully annotated with a uniform set of semantic types. For multiple corpora, this results in multiple models, each of which covers only a small slice of the semantic space. In contrast to this standard setup, there has been little focus on training event extraction systems on multiple corpora. This holds also for the BioNLP STs, even though they are open challenges that explicitly allow external resources to be used in addition to the given training data.
Several previous studies have combined multiple corpora for domain-specific information extraction tasks, such as named entity (NE) recognition for NE normalisation  and protein-protein interaction (PPI) extraction [10, 11]. However, the corpora combined in these studies contain differences not only in their annotation scopes but also in the definitions of particular semantic types. To explore the reasons for the incompatibility that exists between the semantic types in the different corpora, detailed analyses have been performed on the differences among gene and protein annotations in three common corpora , and on the differences among PPI annotations in five PPI corpora . Despite such manual efforts to identify incompatibilities, no general automatic methods for resolving them have been introduced. Consequently, practical efforts to combine these resources rarely involve methods beyond direct corpus merging, which can show reduced performance compared to training on a single corpus.
There are also many studies on aspects of learning with domain adaptation , multi-task learning  and transfer learning . As these methods make no assumptions regarding the compatibility of annotations, they can also be applied in our setting of multiple corpora with partial overlap. These types of approaches have been employed by several event extraction systems, and they have been demonstrated to improve system performance [17-21]. However, whilst assuring applicability to a wide range of problems, the lack of assumptions regarding the compatibility of annotations between different corpora also prevents them from benefiting from direct combinations of data for training. These methods also limit the scope of the resulting systems to the semantic types annotated within a single target corpus, rather than allowing the extraction of the union of types annotated in the applied corpora.
This paper focusses on the construction of a wide-coverage event extraction system by leveraging multiple corpora with partially overlapping semantic annotations as training data. In contrast to established approaches such as stacking  and simple domain adaptation , we introduce a general method that allows a single model to be trained through the merging of multiple corpora. The single model has a wide-coverage, i.e., it covers all the semantic types that appear in the multiple corpora. The method has the advantage of directly combining annotations of semantic types that are shared across different corpora, whilst also explicitly addressing the potentially negative effect of inconsistent annotation in the merged set of types that are specific only to certain corpora. Our approach is straightforward to implement and can be applied to various machine learning and information extraction tasks.
To evaluate the proposed method, we implement various approaches to corpus combination, integrate these with our existing event extraction system, EventMine , and perform experiments using each approach on seven biomedical event extraction corpora. Our results demonstrate that the combination of partially overlapping corpora can improve extraction performance, and that the best combination strategies can be used to train a general, wide-coverage event extraction system that outperforms systems trained on single corpora. We additionally show that using our approach, EventMine can outperform all previously proposed methods on two benchmark tasks established by the BioNLP ST 2011, the Epigenetics and Post-translational Modifications (EPI) and Infectious Diseases (ID) tasks . Detailed evaluation indicates that the system can benefit from the availability of multiple corpora, not only due to a greater number of instances of shared semantic types, but also by using instances of non-shared types, which can serve as constraints in learning. The ability to learn from multiple corpora also suggests that the use of existing resources can reduce the need for the manual annotation of existing semantic types in new corpora and thus allow more efficient division of labour in annotation tasks.
Our focus here is on the construction of a wide-coverage extraction system from multiple corpora that partially overlap in their annotation scopes, most sharing only a small number of annotated types with the other corpora in the set. In this study, we follow the BioNLP ST task setup, in which named entity annotations (e.g., Protein) are provided to the extraction system as part of its input, and the system aims to extract event structures that involve these named entities from text.
Event annotated corpora
Statistics for training and development portions of applied corpora
Named entity types in applied corpora
Named entity types
Protein, chemical, organism, Regulon-operon,
Protein, Drug, ion, simple molecule, tag
Protein, drug or compound, cellular component, cell, tissue,
organ, anatomical system, organism, [ …]
Statistics for transferable events between training and development portions of applied corpora
Event extraction system: EventMine
The classifiers of each module are constructed as follows.
The trigger/entity detector
constructs a trigger/entity word dictionary from the training data and optional external dictionaries, selects trigger/entity candidate words from all the words in texts using dictionary matching, and builds classifiers using the word candidates as instances and their types (and the negative label NONE) as labels. Note that this dictionary matching is used both in the training and prediction to reduce the computational cost, and this is separated from our filtering method which will be presented in the next section. Features for word candidates include character n-grams, context word n-grams and the shortest dependency paths between word candidates and named entities.
The argument detector
identifies relations between trigger words detected by the trigger detector and their candidate role arguments, and builds classifiers with relations as instances, and role types and NONE as labels. Features include character n-grams in candidate participant text spans (triggers and entities), context word n-grams around the candidate participants, shortest paths between the participants, and shortest paths between the participants and other triggers or entities.
The multiple argument detector
constructs candidate event structures by enumerating all possible combinations of the detected relations, and builds classifiers with candidate structures as instances, and event types and NONE as labels. Features are derived from the participant relations of the candidate event and other relations that include the same participants.
The hedge detector
builds a hedge classifier with events as instances and their hedge types (Negation, Speculation and NONE in the resources considered here) as labels. Features include dependency paths containing the event trigger and relation features derived from participant relations.
Candidate construction is performed identically in training and prediction. For further details on EventMine, please see .
We note that despite substantial differences in implementation details, this general architecture and approach to the integration of machine learning-based classifiers is fairly common in state-of-the-art event extraction systems (e.g., [8, 29]). Our approach is thus directly applicable to systems other than EventMine, and similarly it is expected that our results can be generalised to a number of other systems.
Learning from multiple partially overlapping corpora
In conventional applications of statistical classification-based systems such as EventMine, candidate instances not annotated in the source corpus generate negative examples for training (closed-world assumption). Direct application of a system developed for training on a single corpus to multiple, partially overlapping corpora can thus lead to the creation of spurious negative instances from one corpus for cases that correspond to positive instances in terms of the scope of another corpus.
To avoid such negative effects from the inconsistent annotation of types that are not shared across corpora, whilst also benefiting from the annotation of shared types, it is necessary to generate from each corpus only those training instances that are relevant to types annotated in that corpus. In the setting considered here, this is straightforward - in effect automatic - for positive instances, since positive instances are always explicitly annotated. By contrast, the generation of negative examples must be restricted in a way that differentiates between valid and spurious negatives.
If there were a method to reliably determine whether a candidate instance in one corpus would have been annotated as positive under the criteria of another corpus, we could restrict the generation of negative instances in precisely the correct cases. Although there is no such general, precise, automatic method, we can automatically construct reasonably reliable filtering heuristics in the following way. For text span classification tasks (such as entity/trigger detection in event extraction), we can limit the generation of candidate negative instances in each corpus of the merged data set to only those cases in which the surface expression (or, e.g., its base form) matches at least one positive instance of an annotated type in any corpus that shares the type. For example, it is reasonable to assume that human never appears among the positive instances in the corpus A of the above example, and so none of the instances of the word human that appear in this corpus will be treated as negative instances. Analogously, for relation/event extraction, we can restrict generation to those negatives where the combination of the semantic types of the participants appears labelled as a positive instance of a type in scope of the corpus. For example, corpus A would not contain any Protein-Organism relations, and so no negative instances of this relation type would be generated for the corpus A. In cases where there are no semantic types for the participants, their surface expressions can be used instead, although this restriction may not work well when the surface expressions are diverse.
When applying a filtering approach of this type, adding new, partially overlapping corpora to the set used for training has the benefit of increasing both the positive examples of overlapping semantic types, as well as increasing the set of negative instances for these types.
Extract a set of positive (annotated) instances P j from all the corpora for each type T j .
Construct a filter F j (e.g., a list of surface expressions for text span classification) for each type T j using the set of positive instances P j .
Construct a filter F i for each corpus C i from the filters for all the types annotated in C i
- 4.Extract a set of negative instances N i for each corpus C i
Extract negative instances N i′ for each corpus C i
Select N i by filtering out (likely) spurious negative instances from N i′ using the filter F i (e.g., filter out instances if the surface expressions are not in the list generated above).
- 5.Train a binary classifier for each type T j
Merge N i for all T j -annotated corpora and all remaining positive instances for other types, i.e., P m for all types T m (m≠j), as negative instances N j
Train a binary classifier on the positive instances P j and the negative instances N j
- 6.Train a binary classifier for a negative type
Merge N i for all corpora as positive instances P neg
Merge P j for all types as negative instances N neg
Train a binary classifier on the positive instances P neg and the negative instances N neg
Training must be performed in a one-vs-rest setting separately for each type (instead of a multi-class setting) as the creation of negative instances varies by type. Prediction is then performed without any filtering so that the model can extract all the semantic types in all the target corpora.
Event extraction from multiple partially overlapping corpora
In training the EventMine models using multiple corpora with the filtering approach, we train separate one-versus-rest models for each type, as the set of negative examples that can be applied (that are not filtered) varies depending on the type under consideration, as mentioned previously. In addition to the outlined benefits of our approach in isolated classification settings, we note that in pipeline architectures such as EventMine, the use of merged sets of corpora is also expected to have cumulative benefits in terms of the ability to make use of additional features from instances predicted by preceding modules.
We employed the seven corpora introduced above. We used the standard train/development/test data splits provided for the GE, EPI, ID, and MLEE corpora. The DNAm and EPTM corpora only define train/evaluation splits, which were applied in all experiments on these corpora. As the mTOR corpus is not provided with a specific way to divide the data, we split the corpus into random train and evaluation sets on the document level.
We unified certain aspects of the corpus annotation that did not feature explicit direct overlap so that we can evaluate the results with the same criteria applied in the BioNLP ST 2011. The Catalysis event type in mTOR was replaced with the Positive regulation type, as the Positive regulation type in the other corpora is compatible with the combination of those event types in mTOR . We also replaced DNA domain or region or Protein domain or region in MLEE with the genetic Entity type applied in the other resources . We unified minor, semantically non-significant differences in role type names (e.g., toLoc →ToLoc, Theme1 →Theme) among the corpora. We also filtered out rare role types (removing fromLoc and product from mTOR and MLEE) since they do not appear in all corpora and the arguments of the role types are not Protein.
Finally, we removed event structures appearing only once in the training data (e.g., Binding with eight Themes in mTOR) to reduce computational cost.
Characteristics of compared methods
Learning from multiple corpora
Single corpus-independent model
Filtering falsely created instances
Event extraction settings
We follow the settings used by  for EventMine, with the following four exceptions. Firstly, we do not employ the protein-specific coreference resolver to avoid negative effects on named entities other than Protein. The extension of the coreference resolver to other named entity types is left as future work.
Secondly, we do not employ cross validation in training, to avoid negative effects that cross validation tends to ignore rare event types with small numbers of training instances.
Thirdly, we treat all post-translational modification (PTM) types as a single type in modules other than the trigger detector, which assigns the final type. This is a straightforward extension of the previously proposed EPI setting. Finally, in addition to several lexical and semantic resources, we use meta-knowledge cues from the GENIA meta-knowledge corpus [30, 31] in the hedge detector. For brevity, we do not explain the details of the settings; we refer the reader instead to .
Evaluation on each corpus is performed after removing from the system output all the events involving event types not within the annotation scope of that corpus. We evaluate the results with the official evaluation tools and servers of the BioNLP ST 2011, and we mainly report results for the primary (“FULL”) evaluation criteria, which evaluate both whole event structures and their hedge types.
Results and discussion
Performance on annotated events
Recall / precision / F-scores on the development portions of all the corpora
F-scores on the development portions of the corpora
In the detailed results separated by corpus (Table 6), we find that the performance on GE and MLEE using the naïvely merged corpora (Merge) shows a significant degradation compared to the single corpus results, although an improvement was observed for some of the other corpora. This indicates that the negative effect of simple corpus merging depends on the specific case considered. Our new method (Multiple) and Stacking are the only approaches that consistently improve performance over the Single setting. The stable improvement of Stacking is expected, as stacking trains a model on each target corpus and only adds new information compared to Single. The stable improvement of Multiple shows that the proposed filtering approach was effective in reducing the detrimental effects of spuriously created negative examples. Furthermore, the Multiple setting performed better than Stacking on 6 out of 7 corpora, which shows that direct instance addition in Multiple is usually better than the indirect use of information through stacking. EasyAdapt achieved good performance on ID and MLEE and showed comparable performance to Multiple on most of the corpora. In summary, the Multiple setting achieved the best results on GE, EPI, DNAm and mTOR, the EasyAdapt setting performed best on ID and MLEE, and the Merge setting performed best on EPTM. Taken together, the results in Tables 3 and 6 also show that the number of shared events does not necessarily correlate with the improvement achieved by the Multiple setting, even if the ratio of the number of increased events to the number of original events is quite large, such as for mTOR (Table 3). This effect is not unexpected, considering the various challenges of extracting each event type and the diverse distribution of the shared event types.
F-scores on isolated and overlapping types on the development portions of all the corpora
Isolated (excluding PTM)
Recall / precision / F-scores on the test portions of the corpora
Although the performance differences between EasyAdapt and Multiple are not significant, there is a very important difference between the systems resulting from training using these two approaches. The Multiple setting produces a single system that operates on all the corpora, while the other most successful settings, i.e., Stacking and EasyAdapt, are corpus dependent (Table 4). Stacking produces multiple models, each specialised to a target corpus. EasyAdapt produces a single model, but the features for each instance depend on the target corpus. The consistent improvement demonstrated by our new approach is particularly notable considering that it results in a single, coherent system. The ability to produce such a system is important for practical applications, since it means that the simple application of one system can produce coherent analyses over a large part of the semantic space and avoids the need to combine system outputs for further processing.
We have shown here that the Multiple setting can produce a single, coherent system with highly competitive performance, but the approach still has some limitations. The approach is not as general as methods such as stacking, since it requires task-dependent filters. To further reduce the requirements to apply the method, we can consider ways to easily define filters appropriate for specific tasks, e.g., by providing templates. The setting can filter out correct negative examples, as shown in Figure 6, but we could further incorporate thesauri and dictionaries to extend the filtering dictionaries and relax the filtering. We have treated event types as mutually-exclusive, but event types in different corpora could also overlap or be hierarchically related. These cases would need to be treated separately. To further improve the approach, we thus need to consider ways to extend our method to be able to treat relations among types. For example, we could remove positive examples of overlapping types from negative examples, and merge more specific (child) positive instances to more general (parent) positive instances in a type hierarchy.
Comparison with other event extraction systems
Recall / precision / F-scores on the test portions of BioNLP ST 2011 corpora
We also note that other systems use three corpora at most, while our system uses seven corpora. Nevertheless, the results shown in Table 9 are encouraging, especially since our system with the Multiple setting is a single system, while other systems are separately tuned to each target corpus. The results further show that the single system built with the Multiple setting performs better than the highest published results on both of two established benchmark tasks ID and EPI.
Analysis on additional event types
Manual evaluation results on 261 event instances out of annotation scope
Correct (Strict match)
Acceptable (Loose match)
This paper has presented an approach to the construction of a wide coverage information extraction system through training on multiple corpora with partially overlapping annotation scopes. The approach heuristically detects and filters out unannotated parts from each corpus in an automated manner, so as to reduce the generation of spurious negative instances of types that are specific to particular corpora. The remaining instances are directly combined to make full use of the available annotated information. The approach was implemented in the EventMine system and evaluated on seven corpora, including three established BioNLP Shared Task (ST) resources. For all seven corpora, our approach improved the performance of the system sufficiently to achieve results better than those of systems trained on individual corpora. Our approach overcomes the problem of inconsistent annotation scopes in different corpora, and achieves comparable or higher performance than domain adaptation methods that produce separate models for each corpus. Training using the approach proposed in this paper produces a single system, and thus eliminates the need to combine results produced by multiple systems, each trained on a single corpus. Evaluation on BioNLP ST 2011 data showed that our system outperforms previously published systems on two out of three considered tasks. The wide-coverage event extraction system is made available as both a demo and a RESTful web service . In future work, we will apply this system to the entire set of PubMed articles to enrich domain applications that make use of event extraction results, such as semantic search engines. This task is comparatively straightforward as the resulting system is not restricted to any individual corpus but can identify all targeted events in a single application. We will also extend and apply the method to other statistical NLP tasks such as named entity recognition.
This work is supported by the Biotechnology and Biological Sciences Research Council (BBSRC) [BB/G013160/1]. We thank Paul Thompson for his useful comments.
- Ananiadou S, Pyysalo S, Tsujii J, Kell DB: Event extraction for systems biology by text mining the literature. Trends Biotechnol. 2010, 28 (7): 381-390. 10.1016/j.tibtech.2010.04.005.View ArticlePubMedGoogle Scholar
- Kim JD, Ohta T, Pyysalo S, Kano Y, Tsujii J: Extracting bio-molecular events from literature - the BioNLP’09 shared task. Comput Intell. 2011, 27 (4): 513-540. 10.1111/j.1467-8640.2011.00398.x.View ArticleGoogle Scholar
- Kim JD, Nguyen N, Wang Y, Tsujii J, Takagi T, Yonezawa A: The genia event and protein coreference tasks of the BioNLP shared task 2011. BMC Bioinformatics. 2012, 13 (Suppl 11): S1-10.1186/1471-2105-13-S11-S1.PubMed CentralView ArticlePubMedGoogle Scholar
- Pyysalo S, Ohta T, Rak R, Sullivan D, Mao C, Wang C, Sobral B, Tsujii J, Ananiadou S: Overview of the ID, EPI and REL tasks of BioNLP Shared Task 2011. BMC Bioinformatics. 2012, 13 (Suppl 11): S2-10.1186/1471-2105-13-S11-S2.PubMed CentralView ArticlePubMedGoogle Scholar
- Ohta T, Pyysalo S, Miwa M, Tsujii J: Event extraction for DNA methylation. J Biomed Semantics. 2011, 2 (Suppl 5): S2-10.1186/2041-1480-2-S5-S2.PubMed CentralView ArticlePubMedGoogle Scholar
- Pyysalo S, Ohta T, Miwa M, Tsujii J: Towards exhaustive event extraction for protein modifications. Proceedings of BioNLP’11. 2011, Portland: ACL, 114-123.Google Scholar
- Ohta T, Pyysalo S, Tsujii J: From pathways to Biomolecular events: opportunities and challenges. Proceedings of BioNLP’11. 2011, Portland: ACL, 105-113.Google Scholar
- Björne J, Ginter F, Salakoski T: University of Turku in the BioNLP’11 shared task. BMC Bioinformatics. 2012, 13 (Suppl 11): S4-10.1186/1471-2105-13-S11-S4.PubMed CentralView ArticlePubMedGoogle Scholar
- Wermter J, Tomanek K, Hahn U: High-performance gene name normalization with GeNo. Bioinformatics. 2009, 25 (6): 815-821. 10.1093/bioinformatics/btp071.View ArticlePubMedGoogle Scholar
- Landeghem SV, Saeys Y, Baets BD, de Peer YV: Extracting protein-protein interactions from text using rich feature vectors and feature selection. Proceedings of SMBM’08. Turku:. 2008, Turku Centre for Computer Science (TUCS), 77-84.Google Scholar
- Tikk D, Thomas P, Palaga P, Hakenberg J, Leser U: A comprehensive benchmark of kernel methods to extract protein-protein interactions from literature. PLoS Comput Biol. 2010, 6 (7): e1000837-10.1371/journal.pcbi.1000837.PubMed CentralView ArticlePubMedGoogle Scholar
- Wang Y, Kim JD, Saetre R, Pyysalo S, Tsujii J: Investigating heterogeneous protein annotations toward cross-corpora utilization. BMC Bioinformatics. 2009, 10: 403-10.1186/1471-2105-10-403.PubMed CentralView ArticlePubMedGoogle Scholar
- Pyysalo S, Airola A, Heimonen J, Björne J, Ginter F, Salakoski T: Comparative analysis of five protein-protein interaction corpora. BMC Bioinformatics. 2008, 9 (Suppl 3): S6-10.1186/1471-2105-9-S3-S6.PubMed CentralView ArticlePubMedGoogle Scholar
- Daumé III H, Marcu D: Domain adaptation for statistical classifiers. J Artif Int Res. 2006, 26: 101-126.Google Scholar
- Caruana R: Multitask learning: a knowledge-based source of inductive bias. Proceedings of ICML-1993. 1993, Amherst, MA, USA: Morgan Kaufmann, 41-48.Google Scholar
- Pan SJ, Yang Q: A survey on transfer learning. IEEE Trans Knowl Data Eng. 2010, 22 (10): 1345-1359.View ArticleGoogle Scholar
- Riedel S, McCallum A: Fast and robust joint models for biomedical event extraction. Proceedings of EMNLP’11. 2011, Edinburgh: ACL, 1-12.Google Scholar
- McClosky D, Surdeanu M, Manning C: Event extraction as dependency parsing. Proceedings of ACL-HLT’11. 2011, Portland: ACL, 1626-1635.Google Scholar
- R McGrath L, Domico K, D Corley C, Webb-Robertson BJ: Complex biological event extraction from full text using signatures of linguistic and semantic features. Proceedings of BioNLP Shared Task 2011 Workshop. 2011, Portland: ACL, 130-137.Google Scholar
- Vlachos A, Craven M: Biomedical event extraction from abstracts and full papers using search-based structured prediction. BMC Bioinformatics. 2012, 13 (Suppl 11): S5-10.1186/1471-2105-13-S11-S5.PubMed CentralView ArticlePubMedGoogle Scholar
- Miwa M, Thompson P, Ananiadou S: Boosting automatic event extraction from the literature using domain adaptation and coreference resolution. Bioinformatics. 2012, 28 (13): 1759-1765. 10.1093/bioinformatics/bts237.PubMed CentralView ArticlePubMedGoogle Scholar
- Wolpert DH: Stacked generalization. Neural Netw. 1992, 5: 241-259. 10.1016/S0893-6080(05)80023-1.View ArticleGoogle Scholar
- Daumé III H: Frustratingly easy domain adaptation. Proceedings of ACL’07. 2007, Prague: ACL, 256-263.Google Scholar
- Kim JD, Ohta T, Tsujii J: Corpus annotation for mining biomedical events from literature. BMC Bioinformatics. 2008, 9: 10-10.1186/1471-2105-9-10.PubMed CentralView ArticlePubMedGoogle Scholar
- Caron E, Ghosh S, Matsuoka Y, Ashton-Beaucage D, Therrien M, Lemieux S, Perreault C, Roux P, Kitano H: A comprehensive map of the mTOR signaling network. Mol Syst Biol. 2010, 6: 453-PubMed CentralView ArticlePubMedGoogle Scholar
- Pyysalo S, Ohta T, Miwa M, Cho HC, Tsujii J, Ananiadou S: Event extraction across multiple levels of biological organization. Bioinformatics. 2012, 28 (18): i575—i581-PubMed CentralView ArticlePubMedGoogle Scholar
- Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, Cherry JM, Davis AP, Dolinski K, Dwight SS, Eppig JT, Harris MA, Hill DP, Issel-Tarver L, Kasarskis A, Lewis S, Matese JC, Richardson JE, Ringwald M, Rubin GM, Sherlock G: Gene ontology: tool for the unification of biology. Nat Genet. 2000, 25: 25-29. 10.1038/75556.PubMed CentralView ArticlePubMedGoogle Scholar
- Jiang W, Huang L, Liu Q: Automatic adaptation of annotation standards: Chinese word segmentation and POS tagging - a case study. Proceedings of ACL-IJCNLP’09. 2009, Singapore: ACL, 522-530.View ArticleGoogle Scholar
- Quirk C, Choudhury P, Gamon M, Vanderwende L: MSR-NLP Entry in BioNLP Shared Task 2011. Proceedings of BioNLP Shared Task 2011 Workshop. 2011, Portland: ACL, 155-163.Google Scholar
- Thompson P, Nawaz R, McNaught J, Ananiadou S: Enriching a biomedical event corpus with meta-knowledge annotation. BMC Bioinformatics. 2011, 12: 393-10.1186/1471-2105-12-393.PubMed CentralView ArticlePubMedGoogle Scholar
- Miwa M, Thompson P, McNaught J, Kell D, Ananiadou S: Extracting semantically enriched events from biomedical literature. BMC Bioinformatics. 2012, 13: 108-10.1186/1471-2105-13-108.PubMed CentralView ArticlePubMedGoogle Scholar
- Noreen EW: Computer-Intensive Methods for Testing Hypotheses: An Introduction. 1989, New York: Wiley-InterscienceGoogle Scholar
- McClosky D, Riedel S, Surdeanu M, McCallum A, Manning C: Combining joint models for biomedical event extraction. BMC Bioinformatics. 2012, 13 (Suppl 11): S9-10.1186/1471-2105-13-S11-S9.PubMed CentralView ArticlePubMedGoogle Scholar
- EventMine. [http://www.nactem.ac.uk/EventMine/]
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.