geneCommittee: a web-based tool for extensively testing the discriminatory power of biologically relevant gene sets in microarray data classification

In recent years, there has been growing interest in incorporating expression microarrays as an effective technology for disease diagnosis and prognosis [1, 2]. In this context, the identification and validation of specific gene expression signatures able to discriminate disease classes from microarray data represents a primary field in translational clinical research [3–5]. Consequently, several individual tools and integrative platforms have recently been developed [6–9], giving support to different functionalities related to (i) recognizing optimal gene sets for given data sets, (ii) performing functional analysis from a biological perspective and (iii) automatically building accurate classification models.

In this scenario, the M@CBETH [10] web server [11] implements a two-class microarray classification benchmarking tool able to find the best prediction among different classification methods (support vector machines and Fisher’s discriminant analysis) by using principal component analysis (PCA) to reduce the dimension of the data. Prophet [12] is another web-based tool [13] that automatically builds different classifiers (support vector machines, k-nearest neighbour, diagonal linear discriminant analysis, self-organizing maps and shrunken centroids) and allows them to be used for further sample classification following a leave-one-out cross-validation (LOOCV) schema. The web server supports two ways of ranking genes for subsequent selection (F-ratio and the Wilcoxon statistic) and includes the possibility of performing the functional interpretation of the molecular signature found. More recently, the FiGS [14] on-line workbench [15] builds up diverse gene selection procedures by aligning different feature selection techniques and classifiers. The workbench diversifies the gene selection procedures by incorporating gene clustering options in the feature selection step and different data pre-processing options in the classifier training step. All candidate gene selection procedures are evaluated through the .632+ bootstrap method.

From a different perspective, the Signature Evaluation Tool (SET) [16] provides a standalone application to assist in evaluation of the discrimination power of gene expression signatures. SET implements the Golub’s weighted voting algorithm and the LOOCV schema to distinguish between two supervised groups of samples.

Moreover, several ensemble strategies have also been proposed for microarray data classification [17–20] as a way to exploit multiple methods to discriminate the most relevant clusters from a set of genes. Although these methods are shown to be more robust for pattern detection in microarray data, one problem remains in their results: it is essential to find mechanistic explanations that allow understanding the reasons why a particular set of genes was under or up-regulated. Although the numerical analysis of data seems quite consolidated, its full integration with biological knowledge is still a long way off. In many cases, the signatures discovered by the data classification methods look more like random gene lists than biologically plausible and understandable patterns. In the same line, another shortcoming of standard classification algorithms is that they treat gene expression levels as anonymous attributes. However, a lot is already known about the function and role of many genes in certain biological processes. In this context, the inclusion of additional knowledge sources in the classification process can prevent discovery of the obvious, complement a data-inferred hypothesis with references to already proposed relations, avoid overconfident predictions and allow systematic relation of the analysis findings to present knowledge [21]. To cope with this situation, several authors have proposed the integration and use of external knowledge in ensemble-based predictors. Reboiro-Jato et al. presented genEnsemble [22] a novel classifier able to combine the advantages of ensemble approaches with the benefits obtained from the true integration of biological knowledge in the classification process. The authors demonstrate the robustness of the proposed method when applied to several microarray data in an inter-dataset scenario. Pang et al. combine a random forest (RF) classifier with pathway information to explore the capacity of RF to assess the importance of variables in deriving the enrichment of pathways [23]. In this sense, this work is more focused on the functional analysis than on the classification task. Functional analysis has also been the target for other researchers. FatiGO + [24] is a web-based tool specially oriented to the interpretation of microarray experiments. It uses concepts, such as gene ontology (GO), KEGG pathways, Interpro motifs, Swissprot keywords and text-mining based bioentities related to diseases and chemical compounds, to provide a comprehensive analysis of the results. GeneBrowser [25] is another web-based tool that aims to extract biological knowledge from a list of genes. It combines data from several sources and different visualisation methods to improve biological interpretation and knowledge extraction from a group of genes.

Each of these tools presents unique characteristics supporting specific tasks comprising the whole process of constructing and validating prediction rules based on genes. However, an easy-to-use tool assisting systematic evaluation and interpretation of the discriminant ability of biologically relevant gene sets (i.e.: genes involved in pathways, genes as target of microRNAs, genes in given locations, chemical interacting genes, genes related to syndromes, etc.) is still in high demand by clinical researchers. In this context, we present geneCommittee, an on-line web-based tool giving specific support for extensively testing the discriminatory power of biologically relevant genes in microarray data classification.

The main objective of our geneCommittee server is two-fold: to contribute to the systematic analysis of the discrimination power of genes that are mainly used by wet-lab researchers and practitioners; using available knowledge regarding genes and related biomedical concepts (gene locus, proteins, pathways, diseases, drugs, etc.) to enrich microarrays’ data features for more powerful discrimination of sets of genes. In order to accomplish these goals, custom experiments can be defined using several successful classifiers over different microarray data sets following several validation schemas. To our knowledge, there is no equivalent tool able to facilitate the systematic evaluation of biologically relevant gene sets following a straightforward classification perspective.

geneCommittee is implemented as an AJAX-enabled web application programmed in J2SE 1.6 Java language and it is designed to run on a standard Tomcat 6 Web application server. Its source code is publicly available in Github repository [26] being distributed under the GPL free software license. The ZK development framework [27] was used to construct a rich web user interface with many features of a desktop application.

Architecture

geneCommittee is based on a layered design in which the system was structured in four main tiers: (i) web interface, (ii) controllers, (iii) execution engine, and (iv) data management. Figure 1 shows a block diagram representing geneCommittee architecture and its relationship with other external entities.

As previously mentioned, the web interface uses the ZK framework, which eases the development of desktop-like web applications by providing a set of rich widgets together with an environment that uses intensive AJAX communications between the client and the server. In geneCommittee we have taken advantage of these features to provide a very intuitive user interface to guide the user through the main system workflow.

In the web interface, the application logic is managed by a set of specific controllers. The design of this layer is strongly based on the Model-View-ViewModel (MVVM) architectural pattern supported by the ZK framework, which reduces the coupling between the user interface and the controller classes. Depending on their primary role, geneCommittee defines three different types of controllers: (i) committee controllers, which manage the committee creation workflow, (ii) the diagnostic controller, which is responsible for the classification of new patients, and (iii) the data management controller, which is in charge of data set handling. Additionally, although not represented in Figure 1, other minor controllers handle secondary features like personal data modification, feedback report and help support. As showed in Figure 1, committee and diagnostic controllers use Weka [28] classifiers and feature selection algorithms to build the committees. Additionally, when training a new committee, a special controller will use the web service interface of the GeneBrowser system to enrich a list of previously selected genes.

The execution engine layer is probably the most important piece of geneCommittee, as it contains the core of the application. The execution engine runs almost every task asynchronously. This design principle provides us with two major advantages. On one hand, by having all the tasks running in one single class we can schedule their execution to avoid system overloads. Specifically, geneCommittee application allows the system administrator to select the maximum number of tasks that can be simultaneously executed as a way to control both memory and processor consumption. Moreover, the execution engine automatically interleaves the execution of different user tasks in order to avoid long waits. On the other hand, the execution engine is completely isolated from the user interface, being alive to complete the execution of pending tasks even when the user closes the session. In such a situation, users can request to receive an email when their tasks are finished.

At the bottom part of Figure 1, the data management layer is responsible for guaranteeing data persistence. geneCommittee stores training and test data sets in separate files while the remaining information (e.g. data sets metadata, user data, committees, etc.) is stored in a relational database. All the data is kept in the server at the time it is uploaded or generated, preventing data loss and stopping the user from making unnecessary save actions.

System workflow

With the goal of providing specific support to diagnostic analyses and related clinical management decisions, the workflow implemented in geneCommittee includes 4 distinct phases (see Figure 2): Access control, Data management, Committee training, and Diagnostic. This structure was planned with two different types of users in mind, one with a more statistical background, and another mainly interested in patients’ diagnostic. Nevertheless, any profile between both can easily take advantage of all the implemented functionalities.

In order to gain access to our tool, the first step is to create an account, which will allow each user to keep his own independent workspace. The demo account can also be used, providing a quick overview of the system functionalities. After login, two different paths can be independently followed. The first deals with the management of train datasets and the construction of classification models; the second is related with the exploitation of the developed models to identify specific patients in new datasets.

Committee training

The Committee training area is a key piece in the overall architecture of our geneCommittee server. It implements an easy-to-use and straightforward 6-step wizard for giving specific support to both (i) the initial enrichment of raw datasets and (ii) the later construction, training and validation of interpretable classifiers to finally build an accurate committee of experts. Figure 3 shows all the steps comprising the whole wizard together with the input and output entities stored by gene Committee.

In order to better understand the specific functionality of each block depicted in Figure 3, we briefly introduce every step comprising the Committee training wizard.

S1. Data set

The first action consists of selecting the desired dataset from those previously uploaded using the data management area. For each dataset, general information is shown regarding conditions and samples. Since the selected dataset will be subsequently used along the remaining pipeline, any change in this step will imply that the training process already performed with the actual dataset will be lost unless the whole workflow is completed and saved.

S2. Gene set

In this step the user can select those genes showing higher discriminative potential by using several well-established ranking methods: chi-squared distribution, information gain split method, gain ratio and the relief-f feature filtering algorithm. Additionally, numeric attributes can be converted to binary values and/or missing values merged in order to better adapt raw data to the preferred filtering algorithm.

S3. Enrichment

In this stage of the workflow previously selected genes are automatically enriched using GeneBrowser web services, which provide valuable biological knowledge regarding related enzymes, homologies, ontologies, proteins, pathways, diseases and drugs [25]. All this wealthy data gathered from multiple sources can be used to add new relevant features to be taken into consideration during the rest of the training process. As soon as the on-line gene enrichment process is finished, a new table is generated containing the name of the retrieved items, their sources, the associated p-value, the number of genes involved and a complementary link to GeneBrowser application. This link allows the user to obtain exhaustive information for entries of interest without leaving the geneCommittee workspace.

S4. Classifiers

After the gene set enrichment process, it is necessary to (i) specify and configure the classifiers that will be used and (ii) the desired global evaluation strategy. The global evaluation system implemented in geneCommittee is very flexible, allowing both the use of a single classification method or a specific combination of multiple classifiers. Since each classification algorithm has its own advantages and drawbacks, and taking also in consideration that its performance directly depends on the data, our proposal of having a committee of experts allows choosing the best combination of methods for a specific dataset. Regarding this feature, at the moment our geneCommittee server supports five different types of standard classifiers: (i) k-nearest neighbours, (ii) decision trees, (iii) support vector machines, (iv) naïve Bayes, and (v) random forest. According to the needs, the user can select any number of classifiers (belonging to the same type or not) for which each configuration can be individually established.

S5. Evaluation

Once the list of candidate classifiers is finally defined and the evaluation strategy is set, the evaluation step provides an interactive live visualization of the experiment execution. As soon as the job is finished, the user can choose the desired experts (combination of individual classifiers and gene features) to build a new committee (see Figure 4).

In order to evaluate the performance of each expert when dealing with the initially selected dataset, the statistical analysis carried out by geneCommittee can be conveniently adjusted. Implemented measures include Cohen’s Kappa, accuracy, precision, recall, specificity and F-measure, serving for a better perception of the results’ significance. Another interesting feature is the possibility of visualizing the results of a specific class, or in the case shown in Figure 4, a condition. Following the selection of those classifiers comprising the new generated committee, the user has to save it for further use to evaluate new samples (in Diagnostic mode).

S6. Summary

Finally, the last step included in the Committee training wizard is in charge of showing general information about the input dataset used for training, summarizing the criteria used for performing the gene selection process and introducing some details about the new available committee.

Diagnostic mode

In this area it is possible to directly apply previously trained committees to evaluate new samples. As soon as the user uploads a new dataset corresponding to unseen patient data, the selected committee will start working on the diagnostic of all the samples to identify their corresponding classes (conditions). The whole process carried out by geneCommittee working in Diagnostic mode is showed in Figure 5.

Once the selected committee concludes the processing of the new dataset, geneCommittee presents the result of the classification process as showed in Figure 6.

The information displayed in Figure 6 is structured using a table where each column (except for the first) represents one different patient. In this table, rows are grouped in four main sections:

• Committee: each row contains the diagnostics of one member (expert) of the committee, that is to say, a classifier trained using only the biological information of its associated gene set. Committee members will select one single condition for each patient.

• By Gene Set: this section summarizes the committee member’s diagnostics by grouping the outputs of those experts that share the same gene set. Only the conditions with the highest number of votes are shown.

• By Classifier: in the same way as the previous area, this section groups the committee member’s diagnostics taking into consideration the type of the classifier.

• Voting: this section summarizes the whole diagnostic process carried out by showing the votes that each condition has received, along with a final row evidencing the condition or conditions with the higher number of votes.

The diagnostic view showed in Figure 6 also provides a helpful toolbar with several options for conveniently manage the information displayed.

From the experts list comprising the final committee showed in Table 2, we can clearly observe the association of leukemia and specific biological processes such as positive regulation of phosphatidylinositol 3-kinase cascade [38] or the phagocytosis [36, 39].

When working with geneCommittee it is recommended to include diversity among those gene sets and classifiers used. On one hand, adding different gene sets will expand the search field for geneCommittee to identify those sets containing the most discriminant genes. On the other hand, the variation in the classification schemes is important to get the best experts as, based on the ‘no free lunch theorem’, there is no single classification algorithm capable of classify every different case.

The diagnostic information provided by geneCommittee is one of the most important features of our tool. As each classifier is related with one known biological concept, classification results are easier to understand for non-expert users and, additionally, further conclusions related with these concepts can be extracted. Moreover, the detailed voting for each sample can be used as a simple confidence measurement.

Complementarily, a similar study was designed with our geneCommittee server but using RNA-Seq datasets from the portal of The Cancer Genome Atlas [40]. For this purpose, a data matrix was created with samples related to breast invasive carcinoma (BRCA) disease. Available samples were filtered with the goal of only selecting RNASeqV2 data coming from the IlluminaHiSeq_RNASeqV2 platform of the University of North Carolina (UNC).

As a result, two datasets were created containing normalized gene expression information of the data captured. The first dataset, intended to use for committee training, was composed by 115 samples of the TCGA’s batches 93 and 96. The second dataset, intended to use for testing, was composed by 112 samples of the TCGA’s batches 103 and 105. In both cases, the number of available genes is 20502 with two conditions (i.e.: tumor and normal). These datasets are publicly available in the help menu of our geneCommittee server.

Using our geneCommittee server, a diagnostic committee was created with the first dataset using the following straightforward configuration: (i) electing the 300 best genes according to the Chi-square test, (ii) selecting those sets obtained in the enrichment step for guaranteeing a p-value < 0.0001 and a minimum coverage of 150 and (iii) choosing those classifiers (trained with default parameters) with a kappa value > = 0.95 in the evaluation step. This resulted in a final committee composed of 34 experts.

By using the second dataset for testing purposes, the diagnostic performance of the previously generated committee was remarkable: from the 112 testing samples, and using the unweighted majority vote approach, only two samples were misclassified and one of them with a tie between the two classes.

Currently, our geneCommittee server has been successfully tested in Internet Explorer 8, Firefox 16, Opera 9.62 and Safari 3 browsers working on Windows XP/Vista/7, Ubuntu Linux 10.04 version and Mac OSX 10.5 of Intel architecture.

Nowadays, expression microarrays represent a sound technology for disease diagnosis and prognosis. However, its full exploitation depends greatly on the success of statistical approaches that are able to extract accurate evidences from raw data. Another major challenge in the area is to provide end-user tools that allow them to test hypothesis and to make diagnostics in a seamless environment that hides the complexity of statistical software packages. This paper introduces geneCommittee, a web based application that: a) integrates a set of well-known classification methods in an ensemble framework; b) enables the enrichment of original dataset variables with biologically relevant features; c) provides training classification solutions that allow to select classifiers based on the biological relevance and/or discriminative power; and d) permits applying the experts, a mix of classifier/variable, to diagnose new patients.

In our work, geneCommittee was tested using two different gene expression datasets (i.e.: DNA microarray and RNA-Seq). In both cases, our server has demonstrated a good performance which encourages its suitability for analysing data coming from different gene expression data sources.

The main innovation of geneCommittee is related with its ability to combine the advantages of using multiple complementary classifications algorithms with the benefits obtained from the true integration of biological knowledge in the classification process. From this perspective, our geneCommittee server deals with two main prerequisites needed for the development of valuable translational microarray-based diagnostic tools: (i) a solid understanding of the relative strengths and weaknesses of underlying classification methods and (ii) a biologically plausible and understandable behaviour of such models from a biological point of view.

Project name: geneCommittee

Project home page: http://sing.ei.uvigo.es/GC/

Project source page: https://github.com/michada/GeneCommittee

Operating system: Platform independent

Programming language: Java

Other requirements: Internet browser: Explorer 8, Firefox 3, Opera 9.62 and Safari 3

License: GNU GPL