Predicting gene function using hierarchical multi-label decision tree ensembles
© Schietgat et al; licensee BioMed Central Ltd. 2010
Received: 8 January 2009
Accepted: 2 January 2010
Published: 2 January 2010
S. cerevisiae, A. thaliana and M. musculus are well-studied organisms in biology and the sequencing of their genomes was completed many years ago. It is still a challenge, however, to develop methods that assign biological functions to the ORFs in these genomes automatically. Different machine learning methods have been proposed to this end, but it remains unclear which method is to be preferred in terms of predictive performance, efficiency and usability.
We study the use of decision tree based models for predicting the multiple functions of ORFs. First, we describe an algorithm for learning hierarchical multi-label decision trees. These can simultaneously predict all the functions of an ORF, while respecting a given hierarchy of gene functions (such as FunCat or GO). We present new results obtained with this algorithm, showing that the trees found by it exhibit clearly better predictive performance than the trees found by previously described methods. Nevertheless, the predictive performance of individual trees is lower than that of some recently proposed statistical learning methods. We show that ensembles of such trees are more accurate than single trees and are competitive with state-of-the-art statistical learning and functional linkage methods. Moreover, the ensemble method is computationally efficient and easy to use.
Our results suggest that decision tree based methods are a state-of-the-art, efficient and easy-to-use approach to ORF function prediction.
The completion of several genome projects in the past decade has generated the full genome sequence of many organisms. Identifying open reading frames (ORFs) in the sequences and assigning biological functions to them has now become a key challenge in modern biology. This last step, which is the focus of our paper, is often guided by automatic discovery processes which interact with the laboratory experiments.
More precisely, machine learning techniques are used to predict gene functions from a predefined set of possible functions (e.g., the functions in the Gene Ontology). Afterwards, the predictions with highest confidence can be tested in the lab. There are two characteristics of the function prediction task that distinguish it from common machine learning tasks: (1) a single gene may have multiple functions; and (2) the functions are organized in a hierarchy: a gene that is related to some function is automatically related to all its ancestor functions (this is called the hierarchy constraint). This particular problem setting is known in machine learning as hierarchical multi-label classification (HMC) and recently, many approaches have been proposed to deal with it [1–19]. These approaches differ with respect to a number of characteristics: which learning algorithm they are based on, whether the hierarchy constraint is always met and whether they can deal with hierarchies structured as a directed acyclic graph (DAG), such as the Gene Ontology, or are restricted to hierarchies structured as a rooted tree, like MIPS's FunCat.
Decision trees are a well-known type of classifiers that can be learned efficiently from large datasets, produce accurate predictions and can lead to knowledge that provides insight in the biology behind the predictions, as demonstrated by Clare et al. . They have been applied to several machine learning tasks . In earlier work , we have investigated how they can be extended to the HMC setting: we presented an HMC decision tree learner that takes into account the hierarchy constraint and that is able to process DAG structured hierarchies.
In this article, we show that our HMC decision tree method outperforms previously published approaches applied to S. cerevisiae and A. thaliana. Our comparisons primarily use precision-recall curves. This evaluation method is well-suited for the HMC tasks considered here, due to the large class skew present in these tasks.
Moreover, we show that by upgrading our method to an ensemble technique, classification performance improves further. Ensemble techniques are learning methods that construct a set of classifiers and classify new data instances by taking a vote over their predictions. Experiments show that ensembles of decision trees outperform Bayesian corrected support vector machines , a statistical learning method for gene function prediction, on S. cerevisiae data, and methods participating in the MouseFunc challenge [21, 22] on M. musculus data.
A number of machine learning approaches have been proposed in the area of functional genomics. They have been applied in the context of gene function prediction in S. cerevisiae, A. thaliana or M. musculus. We have grouped them according to the learning approach they use.
Network based methods
Several approaches predict functions of unannotated genes based on known functions of genes that are nearby in a functional association network or protein-protein interaction network [2, 4, 5, 8, 15–17]. GENE FAS , for example, predicts functions of unannotated yeast genes based on known functions of genes that are nearby in a functional association network. GENE MANIA  calculates per gene function a composite functional association network from multiple networks derived from different genomic and proteomic data sources.
These approaches are based on label propagation and do not return a global predictive model. However, a number of approaches were proposed to combine predictions of functional networks with those of a predictive model. Kim et al.  combine them with predictions from a Naive Bayes classifier. The combination is based on a simple aggregation function. The Funckenstein system  uses logistic regression to combine predictions made by a functional association network with predictions from a random forest.
Kernel based methods
Deng et al.  predict gene functions with Markov random fields using protein interaction data. They learn a model for each gene function separately and ignore the hierarchical relationships between the functions. Lanckriet et al.  represent the data by means of a kernel function and construct support vector machines for each gene function separately. They only predict top-level classes in the hierarchy. Lee et al.  have combined the Markov random field approach of  with the SVM approach of  by computing diffusion kernels and using them in kernel logistic regression.
Obozinski et al.  present a two-step approach in which SVMs are first learned independently for each gene function separately (allowing violations of the hierarchy constraint) and are then reconciliated to enforce the hierarchy constraint. Barutcuoglu et al.  have proposed a similar approach where unthresholded support vector machines are learned for each gene function and then combined using a Bayesian network so that the predictions are consistent with the hierarchical relationships. Guan et al.  extend this method to an ensemble framework that is based on three classifiers: a classifier that learns a single support vector machine for each gene function, the Bayesian corrected combination of support vector machines mentioned above, and a classifier that constructs a single support vector machine per gene function and per data source and forms a Naive Bayes combination over the data sources.
Methods that learn a separate model for each function have several disadvantages. Firstly, they are less efficient, because n models have to be built (with n the number of functions). Secondly, they often learn from strongly skewed class distributions, which is difficult for many learners.
Decision tree based methods
Clare  presents an HMC decision tree method that learns a single tree for predicting gene functions of S. cerevisiae. She adapts the well-known decision tree algorithm C4.5  to cope with the issues introduced by the HMC task. First, where C4.5 normally uses class entropy for choosing the best split, her version uses the sum of entropies of the class variables. Second, she extends the method to predict classes on several levels of the hierarchy, assigning a larger cost to misclassifications higher up in the hierarchy. The resulting tree is transformed into a set of rules, and the best rules are selected, based on a significance test performed on a separate validation set. Note that this last step violates the hierarchy constraint, since rules predicting a class can be dropped while rules predicting its subclasses are kept. The non-hierarchical version of her method was later used to predict GO terms for A. thaliana . Here, the annotations are predicted for each level of the hierarchy separately.
Hayete and Bienkowska  build a decision tree for each GO function separately using information about protein assignments in the same functional domain. As mentioned earlier, methods that learn separate models for each function have several disadvantages. Moreover, Vens et al.  show that in the context of decision trees, separate models are less accurate than a single HMC tree that predicts all functions at once.
Blockeel et al.  present to our knowledge the first decision tree approach to HMC that exploits the given class hierarchy and predicts all classes with a single decision tree. Their method is based on the predictive clustering tree framework . This method was first applied to gene function prediction by Struyf et al. . Later, Blockeel et al.  propose an improved version of the method and evaluate it on yeast functional genomics data. Vens et al.  extend the algorithm towards hierarchies structured as DAGs and show that learning one decision tree for simultaneously predicting all functions outperforms learning one tree per function (even if those trees are built taking into account the hierarchy).
We first discuss the approach to building HMC trees presented in  and then extend it to build ensembles of such trees.
Using predictive clustering trees for HMC tasks
The approach that we present is based on decision trees and is set in the predictive clustering tree (PCT) framework . This framework views a decision tree as a hierarchy of clusters: the top-node corresponds to one cluster containing all training examples, which is recursively partitioned into smaller clusters while moving down the tree. PCTs can be applied to both clustering and prediction tasks. The PCT framework is implemented in the CLUS system, which is available at http://www.cs.kuleuven.be/~dtai/clus.
PCTs  can be constructed with a standard "top-down induction of decision trees" (TDIDT) algorithm, similar to CART  or C4.5 . The algorithm takes as input a set of training instances (i.e., the genes and their annotations). It searches for the best acceptable test that can be put in a node. If such a test can be found then the algorithm creates a new internal node and calls itself recursively to construct a subtree for each subset (cluster) in the partition induced by the test on the training instances. To select the best test, the algorithm scores the tests by the reduction in variance (which is defined below) they induce on the instances. Maximizing variance reduction maximizes cluster homogeneity and improves predictive performance. If no acceptable test can be found, that is, if no test significantly reduces variance (as measured by a statistical F-test), then the algorithm creates a leaf and labels it with a representative case, or prototype, of the given instances.
To apply PCTs to the task of hierarchical multi-label classification, the variance and prototype are defined as follows .
The heuristic for choosing the best test for a node of the tree is to maximize the variance reduction as discussed before, with the above definition of variance. Note that our definition of w(c) allows the classes to be structured in a DAG, as is the case with the Gene Ontology.
Second, a classification tree stores in a leaf the majority class for that leaf; this class will be the tree's prediction for examples arriving in the leaf. But in our case, since an example may have multiple classes, the notion of "majority class" does not apply in a straightforward manner. Instead, the mean of the class vectors of the examples in that leaf is stored. Recall that is the proportion of examples in the leaf belonging to c i . An example arriving in the leaf can therefore be predicted to belong to class c i if is above some threshold t i , which can be chosen by the user. To ensure that the predictions obey the hierarchy constraint (whenever a class is predicted its superclasses are also predicted), it suffices to choose t i ≤ t j whenever c i is a superclass of c j . The PCT in Figure 1 has a threshold of t i = 0.4 for all i.
CLUS-HMC is the instantiation (with the distances and prototypes defined as above) of the PCT algorithm implemented in the CLUS system.
Ensembles of PCTs
Ensemble methods are learning methods that construct a set of classifiers for a given prediction task and classify new examples by combining the predictions of each classifier. In this paper we consider bagging, an ensemble learning technique that has primarily been used in the context of decision trees. In preliminary experiments, we also considered two other ensemble learning techniques: random forests  and an adapted version of the boosting approach for regression trees by Drucker . However, neither method performed better than simple bagging.
Bagging  is an ensemble method where the different classifiers are constructed by making bootstrap replicates of the training set and using each of these replicates to construct one classifier. Each bootstrap sample is obtained by randomly sampling training instances, with replacement, from the original training set, until the sample contains the same number of instances as the original training set. The individual predictions given by each classifier can be combined by taking the average (for numeric targets) or the majority vote (for nominal targets).
Breiman has shown that bagging can give substantial gains in the predictive performance of decision tree learners . Also in the case of learning PCTs for predicting multiple targets at once (multi-task learning ), decision tree methods benefit from the application of bagging . However, it is clear that, by using bagging on top of the PCT algorithm, the learning time of the model increases significantly, resulting in a clear trade-off between predictive performance and efficiency to be considered by the user.
The algorithm for bagging PCTs takes as input the parameter k, denoting the number of trees in the ensemble. In order to make predictions, the average of all class vectors predicted by the k trees in the ensemble is computed, and then the threshold is applied as before. This ensures that the hierarchy constraint holds. We call the resulting instantiation of the bagging algorithm around the CLUS-HMC algorithm CLUS-HMC-ENS.
Results and discussion
How well does CLUS-HMC perform on functional genomics data and what is the improvement, if any, that can be obtained by using CLUS-HMC-ENS on such tasks?
How does the predictive performance of the proposed algorithms compare to results reported in the biomedical literature?
In order to answer these questions, we compare our results to the results reported by Clare and King  and Barutcuoglu et al.  on S. cerevisiae, to the results reported by Clare et al.  on A. thaliana, and to the results of the groups participating in the MouseFunc challenge [21, 22] on M. musculus. The methods used in these studies were discussed in the "Related work" section.
For S. cerevisiae and A. thaliana, the datasets that we use in our evaluation are exactly those datasets that are used in the cited articles. They are available, together with the parameter settings that can be used to reproduce the results, at the following webpage: http://www.cs.kuleuven.be/~dtai/clus/hmc-ens. For M. musculus, the (raw) data is available at http://hugheslab.med.utoronto.ca/supplementary-data/mouseFunc_I/, while the dataset we assembled from it is available at the former webpage.
Next to predicting gene functions of three organisms (S. cerevisiae, A. thaliana, and M. musculus), we consider two annotation schemes in our evaluation: FunCat (developed by MIPS ), which is a tree-structured class hierarchy and the Gene Ontology (GO) , which forms a directed acyclic graph instead of a tree: each term can have multiple parents.
The first dataset we use (D 0 ) was described by Barutcuoglu et al.  and is a combination of different data sources. The input feature vector for a gene consists of pairwise interaction information, membership to colocalization locale, possession of transcription factor binding sites and results from microarray experiments, yielding a dataset with in total 5930 features. The 3465 genes are annotated with function terms from a subset of 105 nodes from the Gene Ontology's biological process hierarchy.
We also use the 12 yeast datasets (D 1 - D 12 ) from . The datasets describe different aspects of the genes in the yeast genome. They include five types of bioinformatics data: sequence statistics, phenotype, secondary structure, homology and expression. The different sources of data highlight different aspects of gene function. The genes are annotated with functions from the FunCat classification schemes. Only annotations from the first four levels are given.
D 1 (seq) records sequence statistics that depend on the amino acid sequence of the protein for which the gene codes. These include amino acid frequency ratios, sequence length, molecular weight and hydrophobicity.
D 2 (pheno) contains phenotype data, which represents the growth or lack of growth of knock-out mutants that are missing the gene in question. The gene is removed or disabled and the resulting organism is grown with a variety of media to determine what the modified organism might be sensitive or resistant to.
D 3 (struc) stores features computed from the secondary structure of the yeast proteins. The secondary structure is not known for all yeast genes; however, it can be predicted from the protein sequence with reasonable accuracy, using Prof . Due to the relational nature of secondary structure data, Clare performed a preprocessing step of relational frequent pattern mining; D3 includes the constructed patterns as binary attributes.
D 4 (hom) includes for each yeast gene, information from other, homologous genes. Homology is usually determined by sequence similarity; here, PSI-BLAST  was used to compare yeast genes both with other yeast genes and with all genes indexed in SwissProt v39. This provided for each yeast gene a list of homologous genes. For each of these, various properties were extracted (keywords, sequence length, names of databases they are listed in, ...). Clare preprocessed this data in a similar way as the secondary structure data to produce binary attributes.
D 5 , ..., D 12 . Many microarray datasets exist for yeast and several of these were used . Attributes for these datasets are real valued, representing fold changes in expression levels.
We use six datasets from , originating from different sources: sequence statistics, expression, predicted SCOP class, predicted secondary structure, InterPro and homology. Each dataset comes in two versions: with annotations from the FunCat classification scheme and from the Gene Ontology's molecular function hierarchy. Again, only annotations for the first four levels are given. We use the manual annotations for both schemes.
D 13 (seq) records sequence statistics in exactly the same way as for S. cerevisiae. D 14 (exprindiv) contains 43 experiments from NASC's Affymetrix service "Affywatch" http://affymetrix.arabidopsis.info/AffyWatch.html, taking the signal, detection call and detection p-values. D 15 (scop) consists of SCOP superfamily class predictions made by the Superfamily server . D 16 (struc) was obtained in the same way as for S. cerevisiae. D 17 (interpro) includes features from several motif or signature finding databases, like PROSITE, PRINTS, Pfam, ProDom, SMART and TIGRFAMs, calculated using the EBI's stand-alone InterProScan package . To obtain features, the relational data was mined in the same manner as the structure data. D 18 (hom) was obtained in the same way as for S. cerevisiae, but now using SwissProt v41.
We use the data that was provided for the MouseFunc challenge [21, 22]. It consists of 21603 genes, of which 1718 are set aside as test genes. Each gene is annotated with GO terms from a specified subset of the Gene Ontology. The annotations are up-propagated using the Gene Ontology's "is-a" and "part-of" relation. The data is composed of several sources: gene expression data, protein sequence pattern annotations, protein-protein interactions, phenotype annotations, phylogenetic profile and disease associations. In order to construct a single dataset (D 19 ), we joined all data tables, removed attributes with fewer than five non-zero values and computed additional attributes that indicate for each gene the classes of other genes to which it is linked through a protein-protein interaction (only considering training set genes). This yields 18746 attributes in total. The resulting representation is similar to the one used by Guan et al. .
We report the performance of the different methods with precision-recall (PR) and ROC  based evaluation measures. This is motivated by the following two observations: (1) both measures have been used before to evaluate approaches to gene function prediction [1, 8, 22], and (2) they both allow to simultaneously compare classifiers for different classification thresholds. Of both measures, PR based evaluation better suits the characteristics of typical HMC datasets, in which many classes are infrequent (i.e., typically only a few genes have a particular function). Viewed as a binary classification task for each class, this implies that for most classes the number of negative instances by far exceeds the number of positive instances. In some cases, it is preferred to recognize the positive instances (i.e., that a gene has a given function), rather than correctly predict the negative ones (i.e., that a gene does not have a particular function). ROC curves are then less suited for this task, exactly because they also reward a learner if it correctly predicts negative instances (giving rise to a low false positive rate). This can present an overly optimistic view of the algorithm's performance . Therefore, unless it is impossible to reconstruct the PR behaviour of the methods we compare to, we report a PR based evaluation.
where i ranges over all functions, T P i is the number of true positives (correctly predicted positive instances) for function i, F P i is the number of false positives (positive predictions that are incorrect) for function i, and F N i is the number of false negatives (positive instances that are incorrectly predicted negative) for function i. Note that these measures ignore the number of correctly predicted negative examples.
A precision-recall curve (PR curve) plots the precision of a model as a function of its recall. We consider two types of PR curves: (1) a function-wise PR curve for a given function i, which plots Precision i versus Recall i , and (2) an average or pooled PR curve, which plots versus and summarizes the performance of the model across all functions.
We construct the PR curves as follows. Remember that every leaf in the tree contains a vector with for each function the probability that the gene is predicted to have this function. When decreasing the prediction threshold t i from 1 to 0, an increasing number of instances is predicted to belong to c i , causing the recall to increase whereas precision may increase or decrease (with normally a tendency to decrease). Thus, a single tree (or an ensemble of trees) with a specific threshold has a single precision and recall, and by varying the threshold a PR curve is obtained. Such curves allow us to evaluate the predictive performance of a model regardless of t. In the end, a domain expert can choose the threshold corresponding to the point on the curve that looks most interesting to him.
Although a PR curve helps in understanding the predictive behaviour of the model, a single performance score is more useful to compare models. A score often used to this end is the area between the PR curve and the recall axis, the so-called "area under the PR curve" (AUPRC). The closer the AUPRC is to 1.0, the better the model is. We consider two measures that are based on this idea, that correspond to the two types of PR curves and that are often reported in the literature: AU( ), the area under the average PR curve, and , the average over all areas under the function-wise PR curves. Note that AU( ) gives more weight to more frequent functions, while considers the importance of every function to be equal.
Parameter settings for CLUS-HMC and CLUS-HMC-ENS
In the experiments, w0, which determines the weights of the different functions in the decision tree heuristic, is set to 0.75 and the number of examples in each decision tree leaf is lower bounded to 5. The parameter k, which denotes the number of trees used in the ensemble, is set to 50. Preliminary experiments show that performance does not strongly depend on the choice of w0 and that it does not significantly increase after k = 50, so the latter value is a good trade-off between performance and runtime. The significance parameter used in the F-test stopping criterion of CLUS-HMC and CLUS-HMC-ENS is tuned on a separate validation set (1/3 of the training data) and optimized out of 6 possible values (0.001, 0.005, 0.01, 0.05, 0.1, 0.125), maximizing the AU( ). The final model is constructed on the entire training set using the selected value of the significance parameter.
We will first investigate if ensembles improve the predictive performance of CLUS-HMC in gene function prediction and if so, quantify this difference. We will then compare CLUS-HMC and CLUS-HMC-ENS against several state-of-the-art systems in gene function prediction. On the one hand, we will compare CLUS-HMC to C4.5H/M [3, 9], because they both build a single decision tree. On the other hand, we will compare CLUS-HMC-ENS to Bayesian-corrected SVMs , a statistical learning approach, on D0, and to the methods that entered the MouseFunc challenge on D19.
The datasets originating from [3, 9] (i.e., datasets D1 to D18) are divided into a training set (2/3) and a test set (1/3). We use exactly the same splits. For dataset D0, we randomly construct a training and test set with the same ratio. For dataset D19, we use the same training and test sets that were used in the MouseFunc challenge.
Comparison between CLUS-HMC and CLUS-HMC-ENS
Comparison between CLUS-HMC and C4.5H/M
We now concentrate on the comparison of the results obtained by our algorithms to those obtained by other decision tree based algorithms. For the datasets that are annotated with FunCat classes (D1 - D18), we will compare to the hierarchical extension of C4.5 , which we will refer to as C4.5H. For the datasets with GO annotations (D13 - D18), we will use the non-hierarchical multi-label extension of C4.5 , as C4.5H cannot handle hierarchies structured as a DAG. We refer to this system as C4.5M.
For their experiments on A. thaliana, Clare et al.  only report results per level of the hierarchy. In order to obtain these results, they learn a separate classifier per level, removing from their training and test set those genes that do not have annotated functions at that level. This approach may give a biased result: when annotating a new gene, it is not known in advance at which levels of the hierarchy it will have functions. Therefore, we reran C4.5M to learn one classifier that uses all training data and tested it on the complete test set.
For evaluating their systems, Clare et al. [3, 9] report precision. Indeed, as the biological experiments required to validate the learned rules are costly, it is important to avoid false positives. However, precision is always traded off by recall: a classifier that predicts one example positive, but misses 1000 other positive examples may have a precision of 1, although it can hardly be called a good classifier. Therefore, we also compute the recall of the models obtained by C4.5H/M. These models were presented as rules for specific classes without any probability scores, so each model corresponds to precisely one point in PR space.
For each of the datasets D1 - D18, these PR points are plotted against the average PR curves for CLUS-HMC. As we are comparing curves with points, we speak of a "win" for CLUS-HMC when its curve is above C4.5H/M's point, and of a "loss" when it is below the point. Under the null hypothesis that both systems perform equally well, we expect as many wins as losses. We observed that only in one case out of 24, for dataset D16 with FunCat annotations, C4.5H/M outperforms CLUS-HMC. For all other cases there is a clear win for CLUS-HMC. Representative PR curves can be found in Figures 4, 5 and 6.
We can conclude that CLUS-HMC is the tree-building system that yields the best predictive performance. Compared with other existing methods, we are able to obtain the same precision with higher recall, or the same recall with higher precision. Moreover, the hierarchy constraint is always fulfilled, which is not the case for C4.5H/M.
Comparing individual rules
Note from Figure 9 that an even higher precision can be obtained with CLUS-HMC-ENS, although the rules which lead to this prediction are more complex.
Comparison between CLUS-HMC-ENS and Bayesian-corrected SVMs
In this section, we compare CLUS-HMC-ENS to the statistical learning method of Barutcuoglu et al. , which consists of Bayesian-corrected SVMs (see "Related work"). We will further refer to this method as BSVM. The authors have used dataset D0 to evaluate their method and report class-wise area under the ROC convex hull (AUROC) for a small subset of 105 nodes of the Gene Ontology. As only AUROC scores are reported by Barutcuoglu et al. , we adopt the same evaluation metric for this comparison.
Barutcuoglu et al.  build a bagging procedure around their system and report out-of-bag error estimates  as evaluation, which removes the need for a set-aside test set. Out-of-bag error estimation proceeds as follows: for each example in the original training set, the predictions are made by aggregating only over those classifiers for which the example was not used for training. This is the out-of-bag classifier. The out-of-bag error estimate is then the error rate of the out-of-bag classifier on the training set. The number of bags used in this procedure was 10. To compare our results, we use exactly the same method.
Moreover, CLUS-HMC-ENS is faster than BSVM. Runtimes are compared for one of the datasets having annotations from Gene Ontology's complete biological process hierarchy (in particular, we used D16, which is annotated with 629 classes). Run on a cluster of AMD Opteron processors (1.8 - 2.4 GHz, ≥ 2 GB RAM), CLUS-HMC-ENS required 15.9 hours, while SVM-light , which is the first step of BSVM, required 190.5 hours for learning the models (i.e., CLUS-HMC-ENS is faster by a factor 12 in this case).
Comparison between CLUS-HMC-ENS and the methods in the MouseFunc challenge
In this section we compare CLUS-HMC-ENS to the seven systems that submitted predictions to the MouseFunc challenge. These systems are the ensemble extension of BSVM  (which we will call BSVM+), Kernel Logistic Regression  (which we will call KLR), calibrated SVMs  (which we will call CSVM), GENE FAS , GENE MANIA , the combined functional network and classifier strategy of Kim et al.  (which we will call KIM) and the Funckenstein system . These methods were described in the "Related work" section. Note that, when comparing the results, one should keep in mind that each team independently constructed a dataset, possibly using different features. As a result, the differences in performance can be due not only to the learning methods compared, but also the different feature sets used by the methods. As mentioned in the "Datasets" section, the representation that we use is the one of the BSVM+ team.
The organizers have made available a program that computes several evaluation measures and was used to compare the results by the different participating teams in the challenge. This software is available at the same URL where the data can be found, and computes AUROC scores and precision values at several levels of recall for a list of GO terms.
A close inspection of this program reveals that it exhibits some undesirable behaviour. This can easily be verified by observing the result for a classifier that always predicts the same value. The correct function-wise PR curve for any GO term would be a straight line parallel to the recall axis, with precision equal to the frequency of the term. However, the PR curve returned by the software differs from this. If the ordering in which the genes are processed happens to start with a positive gene, then the precision at zero recall equals one. Moreover, if the ordering ends with a negative gene, the precision at recall one is still higher than the class frequency. The ordering in which the examples are processed should be independent from the resulting PR curve.
For this reason, we included the computation of precision and recall in the Clus software. Because the MouseFunc website lists a prediction matrix (containing for each gene-term pair the corresponding probability that the gene is annotated with the GO term) for each of the methods we compare to, we can run our own evaluation program on these predictions, producing corrected results for these methods.
Each method gives predictions for 2815 selected GO terms. These terms are divided into 12 disjunct subsets corresponding to all combinations of the three GO branches (Biological Process, Molecular Function and Cellular Component) with four ranges of specificity, which is defined as the number of genes in the training set to which each term is annotated (3-10, 11-30, 31-100 and 101-300). We have adopted the same subsets and trained and evaluated our models on each of them. Since 1846 of the selected 2815 GO terms were used as annotation in the test set, our evaluation of all the systems is based only on those.
Comparison of AU( ) between Clus-HMC-Ens and the MouseFunc systems
Comparison of between CLUS-HMC-ENS and the MouseFunc systems
Comparison of between Clus-HMC-Ens and the MouseFunc systems
The fact that CLUS-HMC-ENS performs better according to AU( ) than to and can be explained as follows. The variance function used to select the best tests gives a higher weight to functions at higher levels of the hierarchy (see "Methods" section), causing CLUS-HMC-ENS to perform well especially on those functions. In contrast to and , which consider each function as equal, the AU( ) evaluation measure shares the idea of giving a higher penalty to mistakes made for functions at higher levels of the hierarchy.
We can conclude that, in general, the performance of CLUS-HMC-ENS is not significantly different from that of BSVM+, which has been evaluated on the same dataset. Moreover, also compared to the other systems, which have used other preprocessing methods, CLUS-HMC-ENS is competitive: only the Funckenstein method and GENE MANIA produce significantly better results on 3 and 2 evaluation measures, respectively. In a function-wise comparison over all 12 subsets (1846 functions in total), CLUS-HMC-ENS still performed better than Funckenstein on 607 (according to AUPRC) and 625 (according to AUROC) functions, while it had an equal score for 98 (AUPRC) and 97 (AUROC) functions. Similarly, it performed better than GENE MANIA on 645/563 functions and had an equal score for 84/88 functions, respectively. This shows that none of the methods is guaranteed to be the best choice for any given function.
This comparison to the methods in the MouseFunc competition suggests that incorporating functional linkage information in the predictions made by an ensemble method can substantially improve its performance. How this could be achieved for CLUS-HMC-ENS will be investigated in further work.
In this article, we have presented the use of a decision tree learner, called CLUS-HMC, in functional genomics. The learner produces a single tree that predicts, for a given gene, its biological functions from a function classification scheme, such as the Gene Ontology. The main contributions of this work are the introduction of the tree-based ensemble learner CLUS-HMC-ENS and empirical evidence showing that this learner outperforms several state-of-the-art methods on S. cerevisiae, A. thaliana and M. musculus datasets.
First, we have shown that CLUS-HMC outperforms an existing decision tree learner (C4.5H/M) w.r.t. predictive performance. Second, we have shown that the predictive performance boost in regular classification tasks obtained by using ensembles, carries over to the hierarchical multi-label classification context, in which the gene function prediction task is set. Third, by constructing an ensemble of CLUS-HMC-trees, our method outperforms a statistical learner based on SVMs for S. cerevisiae, both in predictive performance and in efficiency. Fourth, this ensemble learner is competitive to statistical and network based methods for M. musculus data.
To summarize, CLUS-HMC can give additional biological insight in the predictions. Moreover, CLUS-HMC-ENS yields state-of-the-art quality for gene function prediction. The software implementing these methods is easy to use and available online as open-source software. As such, CLUS-HMC(-ENS) is competitive to the current state-of-the-art systems and therefore, we believe it should be considered for making automated predictions in functional genomics.
Institute for the Promotion of Innovation through Science and Technology in Flanders (IWT-Vlaanderen) to LS. Research Fund K.U.Leuven to CV and JS. Research Foundation - Flanders (FWO-Vlaanderen) to CV and HB. The GOA Probabilistic Logic Learning to CV. The EU funded project IQ (Inductive Queries for Mining Patterns and Models). The authors thank Amanda Clare and Zafer Barutcuoglu for their cooperation. This research was conducted utilizing high performance computational resources provided by the K.U.Leuven http://ludit.kuleuven.be/hpc.
- Deng M, Zhang K, Mehta S, Chen T, Sun F: Prediction of protein function using protein-protein interaction data. Proceedings of the IEEE Computer Society Bioinformatics Conference, IEEE Computer Society 2002, 197–206. full_textView Article
- Troyanskaya O, Dolinski K, Owen A, Altman R, D B: A Bayesian framework for combining heterogeneous data sources for gene function prediction (in Saccharomyces Cerevisiae). Proceedings of the National Academy of Sciences 2003, 100(14):8348–8353. 10.1073/pnas.0832373100View Article
- Clare A, King RD: Predicting gene function in Saccharomyces cerevisiae. Bioinformatics 2003, 19(Suppl 2):ii42–49.View ArticlePubMed
- Chen Y, Xu D: Global protein function annotation through mining genome-scale data in yeast Saccharomyces cerevisiae. Nucleic Acids Research 2004, 32(21):6414–6424. 10.1093/nar/gkh978View ArticlePubMedPubMed Central
- Karaoz U, Murali T, Letovsky S, Zheng Y, Ding C, Cantor C, Kasif S: Whole-genome annotation by using evidence integration in functional-linkage networks. Proceedings of the National Academy of Sciences 2004, 101(9):2888–2893. 10.1073/pnas.0307326101View Article
- Lanckriet GR, Deng M, Cristianini N, Jordan MI, Noble WS: Kernel-based data fusion and its application to protein function prediction in yeast. Proceedings of the Pacific Symposium on Biocomputing 2004, 300–311.
- Hayete B, Bienkowska J: GOTrees: Predicting GO associations from protein domain composition using decision trees. In Pacific Symposium on Biocomputing. Edited by: Altman RB, Jung TA, Klein TE, Dunker AK, Hunter L. World Scientific; 2005:127–138. full_text
- Chua H, Sung W, Wong L: Exploiting indirect neighbours and topological weight to predict protein function from protein-protein interactions. Bioinformatics 2006, 22(13):1623–1630. 10.1093/bioinformatics/btl145View ArticlePubMed
- Clare A, Karwath A, Ougham H, King RD: Functional bioinformatics for Arabidopsis thaliana. Bioinformatics 2006, 22(9):1130–1136. 10.1093/bioinformatics/btl051View ArticlePubMed
- Barutcuoglu Z, Schapire R, Troyanskaya O: Hierarchical multi-label prediction of gene function. Bioinformatics 2006, 22(7):830–836. 10.1093/bioinformatics/btk048View ArticlePubMed
- Cesa-Bianchi N, Gentile C, Zaniboni L: Incremental algorithms for hierarchical classification. Journal of Machine Learning Research 2006, 7: 31–54.
- Rousu J, Saunders C, Szedmak S, Shawe-Taylor J: Kernel-based learning of hierarchical multilabel classification models. Journal of Machine Learning Research 2006, 7: 1601–1626.
- Lee H, Tu Z, Deng M, Sun F, Chen T: Diffusion kernel-based logistic regression models for protein function prediction. OMICS 2006, 10: 40–55. 10.1089/omi.2006.10.40View ArticlePubMed
- Vens C, Struyf J, Schietgat L, Džeroski S, Blockeel H: Decision trees for hierarchical multi-label classification. Machine Learning 2008, 73(2):185–214. 10.1007/s10994-008-5077-3View Article
- Mostafavi S, Ray D, Warde-Farley D, Grouios C, Morris Q: GeneMANIA: a real-time multiple association network integration algorithm for predicting gene function. Genome Biology 2008, 9(Suppl 1):S4. 10.1186/gb-2008-9-s1-s4View ArticlePubMedPubMed Central
- Kim W, Krumpelman C, Marcotte E: Inferring mouse gene functions from genomic-scale data using a combined functional network/classification strategy. Genome Biology 2008, 9(Suppl 1):S5. 10.1186/gb-2008-9-s1-s5View ArticlePubMedPubMed Central
- Tian W, Zhang L, Tasan M, Gibbons F, King O, Park J, Wunderlich Z, Cherry J, Roth F: Combining guilt-by-association and guilt-by-profiling to predict Saccharomyces cerevisiae gene function. Genome Biology 2008, 9(Suppl 1):S7. 10.1186/gb-2008-9-s1-s7View ArticlePubMedPubMed Central
- Guan Y, Myers C, Hess D, Barutcuoglu Z, Caudy A, Troyanskaya O: Predicting gene function in a hierarchical context with an ensemble of classifiers. Genome Biology 2008, 9(Suppl 1):S3. 10.1186/gb-2008-9-s1-s3View ArticlePubMedPubMed Central
- Obozinski G, Lanckriet G, Grant C, Jordan M, Noble W: Consistent probabilistic outputs for protein function prediction. Genome Biology 2008, 9(Suppl 1):S6. 10.1186/gb-2008-9-s1-s6View ArticlePubMedPubMed Central
- Quinlan J: C4.5: Programs for Machine Learning. Morgan Kaufmann series in Machine Learning, Morgan Kaufmann, Springer Netherlands; 1993.
- Hughes T, Roth F: A race through the maze of genomic evidence. Genome Biology 2008, 9(Suppl 1):S1. 10.1186/gb-2008-9-s1-s1View ArticlePubMedPubMed Central
- Pena-Castillo L, Tasan M, Myers C, Lee H, Joshi T, Zhang C, Guan Y, Leone M, A P, Kim W, Krumpelman C, Tian W, Obozinski G, Qi Y, Mostafavi S, Lin G, Berriz G, Gibbons F, Lanckriet G, Qiu J, Grant C, Barutcuoglu Z, Hill D, Warde-Farley D, Grouios C, Ray D, Blake J, Deng M, Jordan M, Noble W, Morris Q, Klein-Seetharaman J, Bar-Joseph Z, Chen T, Sun F, Troyanskaya O, Marcotte E, Xu D, Hughes T, Roth F: A critical assessment of Mus musculus gene function prediction using integrated genomic evidence. Genome Biology 2008, 9(Suppl 1):S2. 10.1186/gb-2008-9-s1-s2View ArticlePubMedPubMed Central
- Clare A: Machine learning and data mining for yeast functional genomics. PhD thesis. University of Wales, Aberystwyth, Computer Science Department; 2003.
- Blockeel H, Bruynooghe M, Džeroski S, Ramon J, Struyf J: Hierarchical multi-classification. Proceedings of the ACM SIGKDD 2002 Workshop on Multi-Relational Data Mining 2002, 21–35.
- Blockeel H, De Raedt L, Ramon J: Top-down induction of clustering trees. Proceedings of the 15th International Conference on Machine Learning 1998, 55–63.
- Struyf J, Džeroski S, Blockeel H, Clare A: Hierarchical multi-classification with predictive clustering trees in functional genomics. In Progress in Artificial Intelligence: 12th Portugese Conference on Artificial Intelligence. Volume 3808. Lecture Notes in Computer Science, Springer; 2005:272–283.View Article
- Blockeel H, Schietgat L, Struyf J, Džeroski S, Clare A: Decision trees for hierarchical multilabel classification: A case study in functional genomics. In Proceedings of the 10th European Conference on Principles and Practice of Knowledge Discovery in Databases. Volume 4213. Lecture Notes in Artificial Intelligence; 2006:18–29.
- Mewes H, Heumann K, Kaps A, Mayer K, Pfeiffer F, Stocker S, Frishman D: MIPS: A database for protein sequences and complete genomes. Nucleic Acids Research 1999, 27: 44–48. 10.1093/nar/27.1.44View ArticlePubMedPubMed Central
- Breiman L, Friedman J, Olshen R, Stone C: Classification and Regression Trees. Belmont: Wadsworth; 1984.
- Breiman L: Random forests. Machine Learning 2001, 45: 5–32. 10.1023/A:1010933404324View Article
- Drucker H: Improving regressors using boosting techniques. Proceedings of the 14th International Conference on Machine Learning 1997, 107–115.
- Breiman L: Bagging predictors. Machine Learning 1996, 24(2):123–140.
- Caruana R: Multitask Learning. Machine Learning 1997, 28: 41–75. 10.1023/A:1007379606734View Article
- Kocev D, Vens C, Struyf J, Džeroski S: Ensembles of multi-objective decision trees. Proceedings of the 18th European Conference on Machine Learning 2007, 624–631.
- Ashburner M, Ball C, Blake J, Botstein D, Butler H, Cherry J, Davis A, Dolinski K, Dwight S, Eppig J, Harris M, Hill D, Issel-Tarver L, Kasarskis A, Lewis S, Matese J, Richardson J, Ringwald M, Rubin G, Sherlock G: Gene Ontology: Tool for the unification of biology. The Gene Ontology Consortium. Nature Genetics 2000, 25: 25–29. 10.1038/75556View ArticlePubMedPubMed Central
- Ouali M, King R: Cascaded multiple classifiers for secondary structure prediction. Protein Science 2000, 9(6):1162–76. 10.1110/ps.9.6.1162View ArticlePubMedPubMed Central
- Altschul S, Madden T, Schaffer A, Zhang J, Zhang Z, Miller W, Lipman D: Gapped BLAST and PSI-BLAST: A new generation of protein database search programs. Nucl Acids Res 1997, 25: 3389–3402. 10.1093/nar/25.17.3389View ArticlePubMedPubMed Central
- Gough J, Karplus K, Hughey R, Chothia C: Assignment of homology to genome sequences using a library of hidden markov models that represent all proteins of known structure. Molecular Biology 2001, 313(4):903–919. 10.1006/jmbi.2001.5080View Article
- Zdobnov E, Apweiler R: InterProScan - an integration platform for the signature-recognition methods in InterPro. Bioinformatics 2001, 17(9):847–848. 10.1093/bioinformatics/17.9.847View ArticlePubMed
- Provost F, Fawcett T: Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. In Proceedings of the Third International Conference on Knowledge Discovery and Data Mining. AAAI Press; 1998:43–48.
- Davis J, Goadrich M: The relationship between precision-recall and ROC curves. Proceedings of the 23rd International Conference on Machine Learning 2006, 233–240. full_text
- Breiman L: Out-of-bag estimation.Technical Report, Statistics Department, University of California; 1996. [http://ftp.stat.berkeley.edu/pub/users/breiman/OOBestimation.ps.Z]
- Wilcoxon F: Individual comparisons by ranking methods. Biometrics 1945, 1: 80–83. 10.2307/3001968View Article
- Joachims T: Making large-scale SVM learning practical. In Advances in Kernel Methods - Support Vector Learning. Edited by: Scholkopf B, Burges C, Smola A. MIT-Press, Cambridge, MA, USA; 1999.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.