- Open Access
Combining heterogeneous data sources for accurate functional annotation of proteins
© Sokolov etal.; licensee BioMed Central Ltd. 2013
- Published: 28 February 2013
Combining heterogeneous sources of data is essential for accurate prediction of protein function. The task is complicated by the fact that while sequence-based features can be readily compared across species, most other data are species-specific. In this paper, we present a multi-view extension to GOstruct, a structured-output framework for function annotation of proteins. The extended framework can learn from disparate data sources, with each data source provided to the framework in the form of a kernel. Our empirical results demonstrate that the multi-view framework is able to utilize all available information, yielding better performance than sequence-based models trained across species and models trained from collections of data within a given species. This version of GOstruct participated in the recent Critical Assessment of Functional Annotations (CAFA) challenge; since then we have significantly improved the natural language processing component of the method, which now provides performance that is on par with that provided by sequence information. The GOstruct framework is available for download at http://strut.sourceforge.net.
- Gene Ontology
- Gene Ontology Term
- Compatibility Function
- Protein Function Prediction
- Dictionary Lookup
The availability of a large variety of genomic data relevant to the task of protein function prediction poses a data integration challenge due to the heterogeneity of the data sources. While features based on sequence can be readily compared across species, most other data are species-specific: protein-protein interactions are probed experimentally in a given species, and the expression of a given gene measured in one set of experiments is difficult to compare meaningfully to expression measured in another species, under possibly different conditions.
In earlier work we have shown the power of modeling Gene Ontology (GO) term prediction as a hierarchical classification problem using a generalization of the binary SVM to structured output spaces, which allows us to directly predict the GO categories associated with a given protein . Our results demonstrated that the GOstruct method achieves state-of-the-art performance on the Mousefunc competition dataset. In this work we generalize the GOstruct method to allow us to combine both species-specific data and cross-species data computed from sequence, using the framework of multi-view learning . The multi-view learning approach learns a separate classifier for each set of features, and inference is performed jointly in order to predict a label. We demonstrate that the multi-view framework is able to utilize all available information, yielding better performance than sequence based models trained across species and models trained from collections of data within a given species. Preliminary results of the multi-view approach using a limited number of data sources were presented elsewhere . This approach achieved state-of-the-art performance in the recent Critical Assessment of Functional Annotations (CAFA) challenge.
In addition to data that is commonly used in prediction of protein function, namely gene expression and protein-protein interactions (PPI), we report the successful use of large-scale data mined from the biomedical literature, and find that it provides a large boost in accuracy. Together with the text mining data, features based on sequence similarity and PPI account for most of the predictor performance.
We examined the tasks of predicting molecular function, biological process and cellular component in isolation. Our empirical results demonstrate that sequence-based data is more suited to inferring molecular function. Conversely, PPI-based classifiers do well in the other two tasks, outperforming predictors based on any other single source of data. Gene expression data and other sequence-based features provide a marginal increase in accuracy.
The Gene Ontology (GO)  is the current standard for annotating function. GO terms belong to three namespaces that describe a gene product's function: its function on the molecular level, the biological processes in which it participates, and its localization to a cellular component. Each namespace is structured as a hierarchy over its set of keywords, where keywords lower in the hierarchy provide greater specificity of description. Since a protein may have multiple functions in each GO namespace, the problem of protein function prediction can be formulated as hierarchical multi-label classification .
For a long time, the predominant approach to inferring GO function for newly sequenced proteins has been transfer of annotation , where annotations are transferred from proteins with known function on the basis of sequence or structural similarity. Many studies have shown the limitations and issues with this approach [7–10]. Nevertheless, a number of methods employ sequence and structural similarity to make functional annotation predictions with varying degrees of accuracy [11–15]. New schemes are still being proposed today, an example being the algorithm by Hamp, et al. that was used in the 2011 CAFA challenge .
The transfer-of-annotation approach operates like a nearest-neighbor classifier, and is unable to effectively deal with today's noisy high-throughput biological data. This has led to the recent development of machine learning approaches that typically address the problem as a set of binary classification problems: whether a protein should be associated with a given GO term (see e.g., ). The issue with breaking the problem up into a collection of binary classification problems is that the predictions made for individual GO terms will not necessarily be consistent with the constraint that if a term is predicted, all its ancestors in the hierarchy should be predicted as well. Therefore, some methods attempt to reconcile the predictions with the hierarchy to produce a set of consistent annotations e.g., using Bayesian networks or logistic regression [5, 17, 18]. Other methods employ inference algorithms on graphs to directly produce a hierarchical label [19, 20]. But the common approach is to forgo the reconciliation step entirely, partly because the predominant approach to measuring prediction accuracy for this problem is on a "per GO term" basis . In this case, the interpretation of potentially conflicting binary predictions is left up to the user.
The biomedical literature is a resource that has been previously explored for protein function prediction, including as the topic of a shared task (). Several of the previous efforts in this area take advantage of machine learning (e.g. [23–25]), typically training a binary classifier for each GO term as in other related work, where the features employed in the models are derived from informative or discriminating words in text associated to a protein. While some of these approaches show promise, each paper also suggests that integration of external data sources would be useful (and arguably necessary) to improve their results.
Function as a structured label
Rather than treating the task as a collection of binary classification problems ("is a particular GO keyword associated with a particular protein?"), the GOstruct method trains a predictor to infer a full set of annotations directly ("what GO keywords are associated with a particular protein?") using the methodology of structured learning . This is accomplished by learning a compatibility function f (x, y) that measures the level of association between a protein x and a vector of annotations y. Inference of anno-tations is then performed by finding the most compatible label with a given protein: ŷ = arg max y f (x, y). An algorithm aimed at directly inferring complex labels such as GO annotations is called a structured-output method. Structured-output methods have been introduced to the field of machine learning fairly recently and span a number of discriminative and probabilistic approaches . The most popular of these is the structured SVM, which shares many of the advantages of its binary counterpart . Structured SVMs have been successfully applied to a variety of problems, including text categorization [27, 28], prediction of disulfide-bond connectivity , and prediction of enzyme function , but are still not as widely used as binary SVMs due to their higher level of conceptual complexity and lack of easy to use software. In what follows we describe the extension of the GOstruct to multi-view classification.
which reduces to the F1-loss  when using a linear kernel.
to ensure consistent contribution across different feature spaces. Multiple sets of features were combined via unweighted kernel summation.
Training and inference
To make inference via Equation (1) feasible we limited the output space to the labels that occur in the training set only, arguing that this allows the classifier to focus on combinations of GO terms that are biologically relevant. We have found that it is possible to perform approximate inference using an efficient dynamic programming algorithm , but experiments have shown that performing inference that way leads to reduced accuracy, further supporting our choice to limit inference to combinations of GO terms observed in the data. We solve the SVM optimization problem in its dual formulation using the working set approach , which starts by optimizing the dual objective with no constraints. The algorithm then alternates between two steps: adding the most violated constraint to the working set, and solving the optimization problem with respect to the working set. The algorithm terminates when any constraint outside of the working set is violated by no more than any constraint in the working set.
where is a subset of all labels that satisfy the hierarchical constraints and have the i th variable set to 1. The subset is defined in a similar fashion, except for the ith variable being set to 0. This score measures the difference in compatibility values between the most compatible label that includes GO term i and the most compatible label that doesn't; higher values of this difference reflect stronger confidence that GO term i is associated with protein x. Given this score, we can generate ROC and precision-recall curves in the usual way; in our results we quantify performance using the area under the ROC curve (AUC), and the precision at a recall level of 20% (P20R).
Evaluation procedure and model selection
Performance was evaluated using five-fold cross-validation on mouse proteins that have species-specific in-formation and valid annotations. Additional proteins, with cross-species features only, were obtained from the external species H. sapiens. In the interest of keeping the run times down, we randomly subsampled the external set down to 5000 proteins for molecular function and cellular component experiments and down to 3000 proteins for biological process experiments. Since sequence information was used, cross-validation folds were randomly selected such that no two proteins from different folds have more than 50% sequence identity. To select appropriate values for the SVM parameter C, we ran nested four-fold cross-validation on the training data. The value of yielded the highest accuracy on the validation set almost universally.
The number of proteins in mouse and human that participated in classifier training and testing, as well as the number of GO terms considered in each namespace.
number of GO terms
We used features based on protein sequence to construct the cross-species view. Protein sequences for all species were retrieved from the UniProt database (http://uniprot.org). In the cases where a gene has multiple splice forms, the longest one was used. Sequence features were extracted as follows, and a linear kernel was used for the input space kernel for the cross-species view.
We represented a protein in terms of its BLAST similarity scores against a database of annotated proteins . We performed all-vs-all BLAST and the output was post-processed by excluding all hits with e-values above 50.0. The remaining e-values were divided by 50.0 to normalize them. Any values below 1e-10 after normalization were brought up to 1e-10. We then use the negative log of the resulting values as features.
Many biological processes are localized to particular cellular compartments. Information about protein localization can, therefore, be indicative of the function those proteins perform . To take advantage of such information, we use the features computed by the WoLF PSORT algorithm .
Transmembrane protein predictions
A large fraction of proteins are embedded in one of the cellular membranes. Transmembrane proteins tend to be associated with certain functions, such as cell adhesion and transport of ions. Therefore information indicating whether a given protein is a transmembrane protein, and more specifically, how many transmembrane domains it has can also be indicative of protein function. For each protein, we estimated the number of transmembrane domains using the TMHMM program , and an indicator variable was associated with each number of transmembrane domains.
K-mer composition of the N and C termini
The N and C termini of a protein contain signals that are important for protein localization, binding and other protein functions . Therefore we computed features that represent the 3-mer composition of 10 amino acid segments in the N and C termini of each protein.
Low complexity regions
Low-complexity regions in proteins are abundant, have an effect on protein function and are not typically captured by standard sequence comparison methods . We scanned each protein with a sliding window of size 20, and a defined the low-complexity segment as the window that contains the smallest number of distinct amino acids. We used the amino acid composition of that segment as features.
We characterize functional similarity within a species using three sources of data: protein-protein interactions (PPI), gene expression, and protein-GO term co-mentions extracted from the biomedical literature.
We extracted M. musculus protein-protein interaction (PPI) data from version 8.3 of the STRING database . A protein is represented by a vector of variables, where component i provides the STRING evidence score of an interaction between protein i and the given protein. Preliminary experiments indicate that the resulting linear kernel performs as well as the more sophisticated diffusion kernel.
Similarity of expression was measured using a linear kernel across a compendium of 14,696 microarray experiments provided by the authors of Platform for Interactive Learning by Genomics Results Mining (PILGRIM) . Experiments using more sophisticated kernels will be provided elsewhere.
Protein-GO term co-mentions
If a protein and a gene ontology term are mentioned in close proximity in a paper, this can be evidence that the corresponding function is associated with the protein. A set of 11.7 million PubMed abstracts, all Medline abstracts on 9/8/2011 that had title and body text, were used to create a protein-GO term co-mention kernel. The abstracts were fed into a natural language processing pipeline based on the BioNLP UIMA resources (http://bionlp-uima.sourceforge.net/) which consists of the following steps: 1) splitting the abstracts into sentences 2) protein name tagging using the LingPipe named entity recognizer (http://alias-i.com/lingpipe) with the CRAFT model  3) Gene Ontology term recognition via dictionary lookup and 4) extraction of protein-GO term co-occurrence at the abstract level. Protein names were mapped to mouse MGI IDs using MGI name dictionary lookup. Assuming only mouse references allowed us to avoid the full gene normalization problem  and fit in well with the other data sources of the species-specific classifier. The MGI ID-GO ID co-occurrence frequency data was used as features. In this data, each protein is characterized by a vector that provides the number of times it co-occurs with each GO term. In preliminary experiments we also explored the use of protein-protein co-occurrences, but found that they actually hurt performance.
Statistics of the co-mention data across GO namespaces.
Number of GO terms mentioned
Number of protein-GO term co-mentions
Number of unique co-mentions
Mean per protein
1 - 1108
1 - 2034
1 - 531
While it is clear from previous research that exact term matching is inadequate for good recall of Gene Ontology terms in text , it is also clear that accurately recognizing Gene Ontology terms is a challenging problem not only due to linguistic variation  but due to variability in term informativeness in the context of the GO itself . Our conservative exact-match approach to recognizing GO terms is highly precise, and its low coverage is likely offset by the large document collection we have considered in this work. Our collection is orders of magnitude larger than previous collections (for instance,  uses 68,337 abstracts for training and the BioCreative data  consisted of 30,000 (full text) documents). Our use of direct protein mentions within a document to relate proteins to GO terms, and aggregated across the corpus as a whole, also differentiates this work from previous efforts that use externally provided protein-text links. In BioCreative, the test data consisted of protein-document pairs in the input and most systems considered only the information within the document(s) provided for a protein rather than any document in the collection that might mention the protein;  associates proteins to text via curated protein-document links in UniProt. This means our methods consider many more implied relationships than other methods.
Comparing classification approaches
Classifier performance in predicting GO terms in mouse, quantified by area under the ROC curve (AUC) and precision at 20% recall (P20R).
The cross-species SVM outperforms the species-specific SVM in molecular function, which is consistent with the literature demonstrating that molecular function annotations are the easiest to infer from sequence . In the other two namespaces the species-specific SVM performs best, with the strongest contribution coming from the PPI data, as discussed below. This suggests that features that describe the functional network within a species are more predictive of biological process and cellular component than sequence-based features.
Contribution from individual sources of data
Classifier performance in predicting GO terms using individual sources of data and some of their combinations using only data from mouse.
Protein-GO term co-mention
PPI + co-mention
PPI + co-mention + expression
In the species-specific view, the PPI kernel yields the highest accuracy, and outperforms all sequence-based predictors in biological function and cellular component namespaces, including the full cross-species SVM from Table 3. This suggests that functional network information, which is the basis for the "guilt by association" approach for function prediction is effective in those two namespaces (we note that the GOstruct framework was shown to outperform guilt by association methods in a comparison on the Mousefunc challenge data .) Furthermore, these features are complementary to the co-mention features, as demonstrated by the strong increase in performance over either kernel by itself when using the combination of the two. A classifier based solely on gene expression data did not fare well by itself. Nevertheless, inclusion of gene expression data provides a marginal increase in performance. Prediction of function from expression is challenging and others have observed poor performance using expression data alone ; we are currently exploring alternative representations that will improve its usefulness.
The top 5 false positive predictions made by GOstruct.
GOstruct Prediction/Current Annotation (if dierent)
Best Supporting Sentence
GO term(s) in Supporting Sentence
GO:0016787 hydrolase activity
We recently demonstrated that human protein tyrosine phosphatase (PTP) L1, a large cytoplasmic phosphatase also known as PTP-BAS/PTPN13/PTP-1E, is a negative
regulator of IGF-1R/IRS-1/Akt path-way in breast cancer cells.
GO:0016787 hydrolase activity/N/A
GO:0016740 transferase activity/N/A
Using this assay system, chloramphenicol acetyltransferase activity directed by the cTNT promoter/upstream region was between two and three orders of magnitude higher in cardiac or skeletal muscle cells than in fibroblast cells, indicating that cis elements responsible for cell-specific expression reside in this region of the cTNT gene. Many Andersen syndrome cases have been associated with loss-of-function mutations in the inward rectifier K(+) channel Kir2.1 encoded by KCNJ2.
GO:0022857 transmembrane transporter activity/GO:0005242 inward rectifier
GO:0022892 substrate-specific transporter activity/GO:0005242 inward rectifier potassium
IRK1, but not GIRK1/GIRK4 channels, showed a marked specificity toward phosphates in the 4,5 head group positions.
Performance comparison on individual GO terms
For further analysis of performance we examined our classifiers in the context of individual GO terms. For each namespaces we wanted to see whether there are trends in performance as a function of the GO term depth, and whether there are certain categories that are particularly easy or difficult to predict. Overall, we observed a slight upward trend, with predictors achieving higher accuracy on terms deeper in the ontologies. This was most pronounced in the biological process namespace.
This paper presented a multi-view extension to the GOstruct structured output protein function prediction framework. We demonstrated the framework's capability to combine multiple heterogeneous sources of data--annotated proteins from multiple species, and species-specific data that includes PPIs, gene expression, and information mined from the biomedical literature--each providing an increase in performance. The empirical results suggest that sequence-based features are more informative of a protein's molecular function, while functional association features from PPI and text mining data provide a stronger contribution for the prediction of biological process and cellular component annotations. Gene expression provided only a marginal increase in performance and we speculate that more sophisticated kernels are needed to extract more meaningful features. Future work includes the design of these kernels as well as framework extensions to make it more scalable to a higher number of species and larger datasets.
This work was funded by NSF grants DBI-0965616 and DBI-0965768. Chris Funk is supported by NIH training grant T15 LM00945102. NICTA is funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Centre of Excellence program.
This article has been published as part of BMC Bioinformatics Volume 14 Supplement 3, 2013: Proceedings of Automated Function Prediction SIG 2011 featuring the CAFA Challenge: Critical Assessment of Function Annotations. The full contents of the supplement are available online at http://www.biomedcentral.com/bmcbioinformatics/supplements/14/S3.
- Sokolov A, Ben-Hur A: Hierarchical classification of Gene Ontology terms using the GOstruct method. Journal of Bioinformatics and Compuational Biology. 2010, 8 (2): 357-376. 10.1142/S0219720010004744.View ArticleGoogle Scholar
- Blum A, Mitchell T: Combining labeled and unlabeled data with co-training. Proceedings of the eleventh annual conference on Computational learning theory. 1998, ACM, 100-Google Scholar
- Sokolov A, Ben-Hur A: Multi-view prediction of protein function. ACM Conference on Bioinformatics, Computational Biology and Biomedicine. 2011Google Scholar
- Gene Ontology Consortium: Gene ontology: tool for the unification of biology. Nat Genet. 2000, 25-9. 25Google Scholar
- Barutcuoglu Z, Schapire R, Troyanskaya O: Hierarchical multi-label prediction of gene function. Bioinformatics. 2006, 22 (7): 830-10.1093/bioinformatics/btk048.View ArticlePubMedGoogle Scholar
- Loewenstein Y, Raimondo D, Redfern O, Watson J, Frishman D, Linial M, Orengo C, Thornton J, Tramontano A: Protein function annotation by homology-based inference. Genome Biology. 2009, 10 (2): 207-10.1186/gb-2009-10-2-207. [http://genomebiology.com/2009/10/2/207]PubMed CentralView ArticlePubMedGoogle Scholar
- Bork P, Koonin EV: Predicting functions from protein sequences - where are the bottlenecks?. Nature Genetics. 1998, 18: 313-318. 10.1038/ng0498-313.View ArticlePubMedGoogle Scholar
- Devos D, Valencia A: Practical limits of function prediction. PROTEINS-NEW YORK. 2000, 41: 98-107.Google Scholar
- Galperin MY, Koonin EV: Sources of systematic error in functional annotation of genomes: domain rearrangement, non-orthologous gene displacement, and operon disruption. In Silico Biology. 1998, 1: 55-67.PubMedGoogle Scholar
- Rost B, Liu J, Nair R, Wrzeszczynski K, Ofran Y: Automatic prediction of protein function. Cellular and Molecular Life Sciences. 2003, 60 (12): 2637-2650. 10.1007/s00018-003-3114-8.View ArticlePubMedGoogle Scholar
- Boeckmann B, Bairoch A, Apweiler R, Blatter M, Estreicher A, Gasteiger E, Martin M, Michoud K, O'Donovan C, Phan I: The SWISS-PROT protein knowledgebase and its supplement TrEMBL in 2003. Nucleic acids research. 2003, 31: 365-10.1093/nar/gkg095.PubMed CentralView ArticlePubMedGoogle Scholar
- Hamp T, Kassner R, Seemayer S, Vicedo E: Nearest-Neighbor Approaches to Predict Protein Function by Homology Inference Alone. Automatic Function Prediction special interest group meeting at ISMB. 2011Google Scholar
- Hennig S, Groth D, Lehrach H: Automated Gene Ontology annotation for anonymous sequence data. Nucleic Acids Research. 2003, 31 (13): 3712-10.1093/nar/gkg582.PubMed CentralView ArticlePubMedGoogle Scholar
- Martin D, Berriman M, Barton G: GOtcha: a new method for prediction of protein function assessed by the annotation of seven genomes. BMC bioinformatics. 2004, 5: 178-10.1186/1471-2105-5-178.PubMed CentralView ArticlePubMedGoogle Scholar
- Zehetner G: OntoBlast function: From sequence similarities directly to potential functional annotations by ontology terms. Nucleic acids research. 2003, 31 (13): 3799-10.1093/nar/gkg555.PubMed CentralView ArticlePubMedGoogle Scholar
- Mostafavi S, Ray D, Warde-Farley D, Grouios C, Morris Q: GeneMANIA: a real-time multiple associationnetwork integration algorithm for predicting gene function. Genome Biology. 2008, 9 (Suppl 1): S4-10.1186/gb-2008-9-s1-s4.PubMed CentralView ArticlePubMedGoogle Scholar
- Guan Y, Myers C, Hess D, Barutcuoglu Z, Caudy A, Troyanskaya O: Predicting gene function in a hierarchical context with an ensemble of classifiers. Genome Biology. 2008, 9 (Suppl 1): S3-10.1186/gb-2008-9-s1-s3.PubMed CentralView ArticlePubMedGoogle Scholar
- Obozinski G, Lanckriet G, Grant C, Jordan M, Noble W: Consistent probabilistic outputs for proteinfunction prediction. Genome Biology. 2008, 9 (Suppl 1): S6-10.1186/gb-2008-9-s1-s6.PubMed CentralView ArticlePubMedGoogle Scholar
- Joslyn C, Mniszewski S, Fulmer A, Heaton G: The gene ontology categorizer. Bioinformatics. 2004, 20 (suppl 1): i169-10.1093/bioinformatics/bth921.View ArticlePubMedGoogle Scholar
- Mostafavi S, Morris Q: Using the Gene Ontology hierarchy when predicting gene function. Conference on Uncertainty in Artificial Intelligence. 2009Google Scholar
- Peña-Castillo L, Tasan M, Myers C, Lee H, Joshi T, Zhang C, Guan Y, Leone M, Pagnani A, Kim W: A criticalassessment of Mus musculus gene function prediction using integrated genomic evidence. Genome Biology. 2008, 9 (Suppl 1): S2-10.1186/gb-2008-9-s1-s2.PubMed CentralView ArticlePubMedGoogle Scholar
- Blaschke C, Leon E, Krallinger M, Valencia A: Evaluation of BioCreAtIvE assessment of task 2. BMC Bioinformatics. 2005, 6 (suppl 1): S16-10.1186/1471-2105-6-S1-S16.PubMed CentralView ArticlePubMedGoogle Scholar
- Ray S, Craven M: Learning Statistical Models for Annotating Proteins with Function Informationusing Biomedical Text. BMC Bioinformatics. 2005, 6 (Suppl 1): S18-10.1186/1471-2105-6-S1-S18.PubMed CentralView ArticlePubMedGoogle Scholar
- Rice S, Nenadic G, Stapley B: Mining protein function from text using term-based support vectormachines. BMC Bioinformatics. 2005, 6 (Suppl 1): S22-10.1186/1471-2105-6-S1-S22.PubMed CentralView ArticlePubMedGoogle Scholar
- Wong A, Shatkay H: Predicting Protein Function using Text Data from the Biomedical Literature. Automatic Function Prediction special interest group meeting at ISMB. 2011Google Scholar
- Bakir G, Hofmann T, Schölkopf B: Predicting structured data. 2007, The MIT PressGoogle Scholar
- Tsochantaridis I, Joachims T, Hofmann T, Altun Y: Large margin methods for structured and interdependent output variables. Journal of Machine Learning Research. 2006, 6 (2): 453-Google Scholar
- Rousu J, Saunders C, Szedmak S, Shawe-Taylor J: Kernel-based learning of hierarchical multilabel classification models. The Journal of Machine Learning Research. 2006, 7: 1601-1626.Google Scholar
- Taskar B, Chatalbashev V, Koller D, Guestrin C: Learning Structured Prediction Models: A Large Margin Approach. Twenty Second International Conference on Machine Learning (ICML05). 2005Google Scholar
- Astikainen K, Holm L, Pitkanen E, Szedmak S, Rousu J: Towards structured output prediction of enzyme. BMC proceedings. 2008, BioMed Central Ltd, 2: S2-Google Scholar
- Clark W, Radivojac P: Analysis of protein function and its prediction from amino acid sequence. Proteins: Structure, Function, and Bioinformatics. 2011, 79 (7): 2086-2096. 10.1002/prot.23029.View ArticleGoogle Scholar
- Sokolov A: Accurate prediction of protein function using GOstruct. PhD thesis. 2011, Colorado State UniversityGoogle Scholar
- Rogers M, Ben-Hur A: The use of Gene Ontology evidence codes in preventing classifier assessmentbias. Bioinformatics. 2009, 25 (9): 1173-10.1093/bioinformatics/btp122.View ArticlePubMedGoogle Scholar
- Altschul S, Gish W, Miller W, Myers E, Lipman D: Basic local alignment search tool. J Mol Biol. 1990, 215 (3): 403-410.View ArticlePubMedGoogle Scholar
- Horton P, Park K, Obayashi T, Nakai K: Protein subcellular localization prediction with WoLF PSORT. Proceedings of the 4th annual Asia Pacific bioinformatics conference APBC06, Taipei, Taiwan. 2006, Citeseer, 39: 48-Google Scholar
- Krogh A, Larsson B, Von Heijne G, Sonnhammer E: Predicting transmembrane protein topology with ahidden markov model: application to complete genomes1. Journal of molecular biology. 2001, 305 (3): 567-580. 10.1006/jmbi.2000.4315.View ArticlePubMedGoogle Scholar
- Bahir I, Linial M: Functional grouping based on signatures in protein termini. Proteins: Structure, Function, and Bioinformatics. 2006, 63 (4): 996-1004. 10.1002/prot.20903.View ArticleGoogle Scholar
- Coletta A, Pinney J, Solís D, Marsh J, Pettifer S, Attwood T: Low-complexity regions within protein sequences have position-dependent roles. BMC systems biology. 2010, 4: 43-10.1186/1752-0509-4-43.PubMed CentralView ArticlePubMedGoogle Scholar
- Jensen L, Kuhn M, Stark M, Charon S, Creevey C, Muller J, Doerks T, Julien P, Roth A, Simonovic M: STRING 8.a global view on proteins and their functional interactions in 630 organisms. Nucleic acids research. 2009, 37 (suppl 1): D412-PubMed CentralView ArticlePubMedGoogle Scholar
- Greene CS, Troyanskaya OG: PILGRM: an interactive data-driven discovery platform for expert biologists. Nucleic acids research. 2011, 39 (Web Server): W368-W374. [http://dx.doi.org/10.1093/nar/gkr440]PubMed CentralView ArticlePubMedGoogle Scholar
- Verspoor KM, Cohen KB, Lanfranchi A, Warner C, Johnson HL, Roeder C, Choi JD, Funk C, Malenkiy Y, Eckert M, Xue N, WAB, Bada M, Palmer M, Hunter LE: A corpus of full-text journal articles is a robust evaluation tool for revealing differences in performance of biomedical natural language processing tools. BMC Bioinformatics.Google Scholar
- Lu Z, Kao H, Wei C, Huang M, Liu J, Kuo C, Hsu C, Tsai R, Dai H, Okazaki N, Cho H, Gerner M, Solt I, Agarwal S, Liu F, Vishnyakova D, Ruch P, Romacker M, Rinaldi F, Bhattacharya S, Srinivasan P, Liu H, Torii M, Matos S, Campos D, Verspoor K, Livingston K, Wilbur W: The gene normalization task in BioCreativeIII. BMC Bioinformatics. 2011, 12 (Suppl 8): S2-10.1186/1471-2105-12-S8-S2.PubMed CentralView ArticlePubMedGoogle Scholar
- Verspoor C, Joslyn C, Papcun G: The Gene Ontology as a Source of Lexical Semantic Knowledge for a Biological Natural Language Processing Application. Proceedings of the SIGIR'03 Workshop on Text Analysis and Search for Bioinformatics. 2003Google Scholar
- Couto F, Silva M, Coutinho P: Finding genomic ontology terms in text using evidence content. BMC Bioinformatics. 2005, 6 (suppl 1): S21-10.1186/1471-2105-6-S1-S21.PubMed CentralView ArticlePubMedGoogle Scholar
- Lanckriet G, Deng M, Cristianini N, Jordan M, Noble W: Kernel-based data fusion and its application to. Proceedings of the Pacific Symposium on Biocomputing. 2004, World Scientific Singapore, 9: 300-311-Google Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.