Mutual information estimation reveals global associations between stimuli and biological processes
© Suzuki et al; licensee BioMed Central Ltd. 2009
Published: 30 January 2009
Although microarray gene expression analysis has become popular, it remains difficult to interpret the biological changes caused by stimuli or variation of conditions. Clustering of genes and associating each group with biological functions are often used methods. However, such methods only detect partial changes within cell processes. Herein, we propose a method for discovering global changes within a cell by associating observed conditions of gene expression with gene functions.
To elucidate the association, we introduce a novel feature selection method called Least-Squares Mutual Information (LSMI), which computes mutual information without density estimaion, and therefore LSMI can detect nonlinear associations within a cell. We demonstrate the effectiveness of LSMI through comparison with existing methods. The results of the application to yeast microarray datasets reveal that non-natural stimuli affect various biological processes, whereas others are no significant relation to specific cell processes. Furthermore, we discover that biological processes can be categorized into four types according to the responses of various stimuli: DNA/RNA metabolism, gene expression, protein metabolism, and protein localization.
We proposed a novel feature selection method called LSMI, and applied LSMI to mining the association between conditions of yeast and biological processes through microarray datasets. In fact, LSMI allows us to elucidate the global organization of cellular process control.
Advances in microarray technologies enable us to explore the comprehensive dynamics of transcription within a cell. The current problem is to extract useful information from a massive dataset. The primarily used approach is clustering. Cluster analysis reveals variations of gene expression and reduces the complexity of large datasets. However, additional methods are necessary to associate genes in each cluster with genetic function using GO term finder , or to understand stimuli related to specific cellular status.
However, these clustering-association strategies cannot detect global cell status changes because of the division of clusters. Some stimuli activate a specific pathway, although others might change overall cellular processes. Understanding the effect of stimuli in cellular processes directly, in this paper, we introduce a novel feature selection method called Least-Squares Mutual Information (LSMI), which selects features using mutual information without density estimation. Mutual information has been utilized to measure distances between gene expressions . To compute the mutual information in existing methods, density estimation or discritization is required. However, the estimation of gene expression is difficult because we have little knowledge about density function of gene expression profile. LSMI offers an analytic-form solution and avoid the estimation.
Feature selection techniques are often used in gene expression analysis . Actually, LSMI has three advantages compared to existing methods: capability of avoiding density estimation which is known to be a hard problem , availability of model selection, and freedom from a strong model assumption. To evaluate the reliability of ranked features using LSMI, we compare receiver operating characteristic (ROC) curves  to those of existing methods: kernel density estimation (KDE) [6, 7], k-nearest neighbor (KNN) , Edgeworth expansion (EDGE) , and Pearson correlation coefficient (PCC). Thereby, we certify that our method has better performance than the existing methods in prediction of gene functions about biological processes. This fact implies that features selected using our method reflect biological processes.
Using the ranked features, we illustrate the associations between stimuli and biological processes according to gene expressions. Results show that stimuli damage essential processes within a cell, causing association with some cellular processes. From the response to stimuli, biological processes are divisible into four categories: DNA/RNA metabolic processes, gene expression, protein metabolic processes, and protein localization.
Approach – mutual information detection
Note that I s vanishes if and only if x and y are independent. The use of MI allows us to detect no correlation stimulus with a specific gene function or process.
Mathematical definitions related to LSMI are provided in the Methods section. LSMI offers an analytic-form solution, which allows us to estimate MI in a computationally very efficiently manner. It is noteworthy that x includes a multi-dimensional vector. In fact, LSMI can handle a group of stimuli, although generic correlation indices such as Pearson correlation between parameters and target value are calculated independently. Therefore, we can elucidate which type of stimulus has no dependency to biological processes using LSMI.
Datasets and feature selection
In this section, we first prepare datasets to show the association between stimuli and biological process, and introduce feature selection using the datasets.
We compute mutual information between gene expression values grouped by stimuli and class of genes' biological processes. As the class, we use biological process terms in Gene Ontology (GO) categorization . We select GO terms associated with more than 800 and less than 2,000 genes because terms having a small number of genes only describe a fraction of the cell status, whereas terms having a large number of genes indicate functions associated with almost all genes in yeast. Actually, GO has a directed acyclic graph (DAG) structure, and each term has child terms. The GO terms are classified into three categories; we use only biological process terms to identify the changes within a cell. Using this method, we select 12 GO terms.
Gene expression profiles
The gene expression profile is the best comprehensive dataset to associate stimuli and biological processes. We use two different microarray datasets. One is of 173 microarray data under stress conditions of various types . We categorize the 173 stress conditions into 29 groups based on the type of condition such as heat shock, oxidizing condition, etc. The other is of 300 microarray data under gene-mutated conditions . We categorize the genes into 146 groups based on associated GO terms. We use only the GO terms which are associated with 1,500 genes or fewer. We also use child terms on a GO layered structure if the term has more than 1,200 genes. When one gene belongs to multiple GO terms, we classify the gene into the the classification whose number of associated genes is smallest. In both profiles, we remove genes whose expression values are obtained from fewer than 30% of all observed conditions. All missing values are filled out by the average of all the expression values.
Feature selection using LSMI
We use a novel feature selection method called LSMI, which is based on MI, to associate stimuli with cellular processes. Here we consider the forward feature-group addition strategy, i.e., a feature-group score between each input feature-group and output cellular process is computed. The top m feature-groups are used for training a classifier. We predict 12 GO terms independently. We randomly choose 500 genes from among 6, 116 genes on the stress condition dataset for feature-group selection and for training a classifier; the rest are used for evaluating the generalization performance. For using the gene-mutated expression dataset, we select 500 genes from among 6, 210 genes. We repeat this trial 10 times. For classification, we use a Gaussian kernel support vector machine (GK-SVM) , where the kernel width is set at the median distance among all samples and the regularization parameter is fixed at C = 10. We explain the efficiency of feature selection of LSMI in the Discussion section.
As shown in this figure, conditions are divided into two groups. Almost all conditions in the upper cluster have higher rank, whereas those in a lower cluster have higher rank only under specific conditions. The conditions in the upper cluster include strong heat shocks, dithiothreitol (DTT) exposure, nitrogen depletion, and diamide treatments, which are non-natural conditions. The result reveals that non-natural conditions change overall cellular processes.
The GO term clusters are divided into three groups: DNA/RNA metabolism (right), localization of protein (middle), and others (left). The leftmost cluster contains bio synthesis, gene expression process, and protein metabolic process. From this figure, nucleic acid metabolism processes are inferred to be independent from amino acid metabolism processes. We will confirm the independence and consider the division of clusters by using other dataset later.
We herein investigate the details of difference among DNA metabolic process, protein metabolic process and localization of proteins. Under an overexpression condition indicated by sign (A) in Fig. 1, DNA/RNA metabolisms show no correlation with expressions of genes belonging to over-expression genes. This finding of no correlation is one advantage of LSMI. The menadione (vitamin K) exposure condition indicated by (B) in Fig. 1 is associated with localization of proteins. Menadione supplementation causes high toxicity; such toxicity might result from the violation of protein localizations.
A common analytical flow of the expression data is first clustering and then associating clusters with GO terms or pathways. Although clustering reduces the complexity of large datasets, the strategy might fail to detect changes of entire genes within a cell such as metabolic processes.
To interpret such gene expression changes, gene set enrichment analysis  has been proposed. This method treats microarrays independently. Therefore, housekeeping genes are often ranked highly. When gene expressions under various conditions are available, our method would show us the better changes of cellular processes because of the comparison between groups of conditions. The module map  gives a global association between a set of genes and a set of conditions. However, this method requires important changes of gene expressions because it uses hypergeometric distributions to compute correlations. Our correlation index is based on MI. Therefore, we can detect nonlinear dependencies with no correlation. An example is depicted in Fig. 3(III).
Relation between existing and proposed MI estimators. If the order of the Edgeworth expansion is regarded as a tuning parameter, model selection of EDGE is expected to be 'Not available'.
In the AUC figures, the higher curves represent better predictions. For example, Fig. 4(a) shows that LSMI is the highest position, which means that LSMI achieves the best performance among the six methods. In Figs. 4(b) and 4(d), KNN(1) and KNN(5), which are denoted by the light blue and dotted light blue lines, have the best performance. However, in Figs. 4(i), (j) and 4(l), averaged AUCs of KNN using numerous groups are high, whereas the AUCs using small and few groups are low. No systematic model selection strategies exist for KNN and therefore KNN would be unreliable in practice. Fig. 4(c) depicts that EDGE, which is indicated by the light green line, has the highest AUC. In fact, EDGE presumes the normal distribution. Consequently, it works well only on a few datasets. From these figures, LSMI indicated by the blue line appears to be the best feature selection method.
We provided a global view of the associations between stimuli and changes of biological processes based on gene expression profiles. The association is generally difficult to use for making models because of nonlinear correlation. To cope with this problem, we introduced a novel feature selection method called LSMI, which uses MI and can be computed efficiently. In comparison to other feature selection methods, LSMI showed better AUCs in prediction of biological process functions. Consequently, our feature selection results would be more reliable than those obtained using the other methods. We calculated the association between stimuli and GO biological process terms using gene expression profiles. The result revealed that the stimuli are categorized into four types: related to DNA/RNA metabolic process, gene expression, protein metabolic process, and protein localization. LSMI enabled us to reveal the global regulation of cellular processes from comprehensive transcription datasets.
Mutual information estimation
A naive approach to estimating MI is to use a KDE [6, 7], i.e., the densities pxy(x, y), px(x), and py(y) are separately estimated from samples and the estimated densities are used for computing MI. The band-width of the kernel functions could be optimized based on likelihood cross-validation (LCV) , so there remains no open tuning parameter in this approach. However, density estimation is known to be a hard problem  and therefore the KDE-based method may not be so effective in practice.
An alternative method involves estimation of entropies using KNN. The KNN-based approach was shown to perform better than KDE , given that the number k is chosen appropriately – a small (large) k yields an estimator with small (large) bias and large (small) variance. However, appropriately determining the value of k is not straightforward in the context of MI estimation.
The solution of LSMI can be computed by simply solving a system of linear equations. Therefore, LSMI is computationally very efficient. Furthermore, a variant of cross-validation (CV) is available for model selection, so the values of tuning parameters such as the regularization parameter and the kernel width can be adaptively determined in an objective manner.
A new MI estimator
In this section, we formulate the MI inference problem as density ratio estimation and propose a new method of estimating the density ratio.
MI inference via density ratio estimation
drawn from a joint distribution with density pxy(x, y). Let us denote the marginal densities of x i and y i by px(x) and py(y), respectively. The goal is to estimate squared-loss MI defined by Eq.(1).
where α= (α1, α2, ..., α b ) ⊤ are parameters to be learned from samples, ⊤ denotes the transpose of a matrix or a vector, and
φ(x, y) = (φ1(x,y), φ2(x, y), ..., φ b (x,y)) ⊤
0 b denotes the b-dimensional vector with all zeros. Note that φ(x, y) could be dependent on the samples , i.e., kernel models are also allowed. We explain how the basis functions φ(x, y) are chosen in the later section.
A least-squares approach to direct density ratio estimation
where I b is the b-dimensional identity matrix.
We call the above method Least-Squares Mutual Information (LSMI). Thanks to the analytic-form solution, the LSMI solution can be computed very efficiently.
Here, we show a non-parametric convergence rate of the solution of the optimization problem (3).
where gi,j:= g(x i , y j ). We assume that the true density ratio function w(x, y) is contained in model and satisfies
w(x , y) <M0 for all (x , y) ∈ D X × D Y .
Then we have the following theorem. Its proof is omitted due to lack of space.
where ||·||2 means the L2(pxpy)-norm and denotes the asymptotic order in probability.
CV for model selection and basis function design
The performance of LSMI depends on the choice of the model, i.e., the basis functions φ(x, y) and the regularization parameter λ. Here we show that model selection can be carried out based on a variant of CV.
δ(y= vℓ) is a indicator function, which is 1 if y= vℓ and 0 otherwise.
In the experiments, we fix the number of basis functions at
b = min(100, n),
and choose the Gaussian width σ and the regularization parameter λ by CV with grid search.
Relation to existing methods
In this section, we discuss the characteristics of existing and proposed approaches.
Kernel density estimator (KDE)
This procedure is repeated for k = 1, 2, ..., K and choose the value of σ such that the average of the hold-out log-likelihood over all k is maximized. Note that the average hold-out log-likelihood is an almost unbiased estimate of the Kullback-Leibler divergence from p(x) to (x), up to an irrelevant constant.
Based on KDE, MI can be approximated by separately estimating the densities pxy(x, y), px(x) and py(y) using . However, density estimation is known to be a hard problem and therefore the KDE-based approach may not be so effective in practice.
k-nearest neighbor method (KNN)
where ψ is the digamma function.
A practical drawback of the KNN-based approach is that the estimation accuracy depends on the value of k and there seems no systematic strategy to choose the value of k appropriately.
Edgeworth expansion (EDGE)
MI can be expressed in terms of the entropies as
I(X, Y) = H(X) + H(Y) - H(X, Y),
Thus MI can be approximated if the entropies above are estimated.
where Hnormal is the entropy of the normal distribution with covariance matrix equal to the target distribution and κi,j,k(1 ≤ i, j, k ≤ d) is the standardized third cumulant of the target distribution. In practice, all the cumulants are estimated from samples.
If the underlying distribution is close to the normal distribution, the above approximation is quite accurate and the EDGE method works very well. However, if the distribution is far from the normal distribution, the approximation error gets large and therefore the EDGE method may be unreliable. In principle, it is possible to include the fourth and even higher cumulants for further reducing the estimation bias. However, this in turn increases the estimation variance; the expansion up to the third cumulants would be reasonable.
This work was partially supported by KAKENHI (Grant-in-Aid for Scientific Research) on Priority Areas "Systems Genomics" from the Ministry of Education, Culture, Sports, Science and Technology of Japan.
T.S. was supported by the JSPS Research Fellowships for Young Scientists.
This article has been published as part of BMC Bioinformatics Volume 10 Supplement 1, 2009: Proceedings of The Seventh Asia Pacific Bioinformatics Conference (APBC) 2009. The full contents of the supplement are available online at http://www.biomedcentral.com/1471-2105/10?issue=S1
- Boyle EI, Weng S, Gollub J, Jin H, Botstein D, Cherry JM, Sherlock G: GO::TermFinder – open source software for accessing Gene Ontology information and finding significantly enriched Gene Ontology terms associated with a list of genes. bioinformatics. 2004, 20: 3710-3715. 10.1093/bioinformatics/bth456.PubMed CentralView ArticlePubMedGoogle Scholar
- Priness I, Maimon O, Ben-Gal I: Evaluation of gene-expression clustering via mutual information distance measure. BMC Bioinformatics. 2007, 8: 111-10.1186/1471-2105-8-111.PubMed CentralView ArticlePubMedGoogle Scholar
- Yvan Saeys II, Larranaga P: A review of feature selection techniques in bioinformatics. bioinformatics. 2007, 23 (19): 2507-2517. 10.1093/bioinformatics/btm344.View ArticlePubMedGoogle Scholar
- Schölkopf B, Smola AJ: Learning with Kernels. 2002, Cambridge, MA: MIT PressGoogle Scholar
- Pepe MS: Evaluation of Medical Tests for Classification and Prediction. 2003, Oxford PressGoogle Scholar
- Silverman BW: Density Estimation for Statistics and Data Analysis. 1986, Chapman & Hall/CRCView ArticleGoogle Scholar
- Fraser AM, Swinney HL: Independent coordinates for strange attractors from mutual information. Physical Review A. 1986, 33 (2): 1134-1140. 10.1103/PhysRevA.33.1134.View ArticlePubMedGoogle Scholar
- Kraskov A, Stögbauer H, Grassberger P: Estimating mutual information. Physical Review E. 2004, 69: 066138-10.1103/PhysRevE.69.066138.View ArticleGoogle Scholar
- Hulle MMV: Edgeworth Approximation of Multivariate Differential Entropy. Neural Computation. 2005, 17 (9): 1903-1910. 10.1162/0899766054323026.View ArticlePubMedGoogle Scholar
- Guyon I, Elisseeff A: An Introduction to Variable Feature Selection. Journal of Machine Learning Research. 2003, 3: 1157-1182. 10.1162/153244303322753616.Google Scholar
- Torkkola K: Feature extraction by non-parametric mutual information maximization. Journal of Machine Learning Research. 2003, 3: 1415-1438. 10.1162/153244303322753742.Google Scholar
- Comon P: Independent Component Analysis, A new concept?. Signal Processing. 1994, 36 (3): 287-314. 10.1016/0165-1684(94)90029-9.View ArticleGoogle Scholar
- Cover TM, Thomas JA: Elements of Information Theory. 1991, N. Y.: John Wiley & Sons, IncView ArticleGoogle Scholar
- Ashburner M, Ball CA, Blake JA, Botstein D, Butler H: Gene ontology: tool for the unification of biology. The Gene Ontology Consortium. Nat Genet. 2000, 25: 25-29. 10.1038/75556.PubMed CentralView ArticlePubMedGoogle Scholar
- Gasch AP, Spellman PT, Kao CM, Carmel-Harel O, Eisen MB, Storz G, Botstein D, Brown PO: Genomic Expression Programs in the Response of Yeast Cells to Environmental Changes. Molecular Biology of the Cell. 2000, 11 (12): 4241-4257.PubMed CentralView ArticlePubMedGoogle Scholar
- Hughes TR, Marton MJ, Jones AR: Functional Discovery via a Compendium of Expression Proiles. Cell. 2000, 102: 109-126. 10.1016/S0092-8674(00)00015-5.View ArticlePubMedGoogle Scholar
- Schlitt B, Palin K, Rung J, Dietmann S, Lappe M, Ukkonen E, Alvis : From Gene Networks to Gene Function. Genome Research. 2003, 13: 2568-2576. 10.1101/gr.1111403.PubMed CentralView ArticlePubMedGoogle Scholar
- Subramanian A, Tamayo P, Mootha VK, Mukherjee S, Ebert BL, Gillette MA, Paulovich A, Pomeroy SL, Golub TR, Lander ES, Mesirov JP: Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proc Natl Acad Sci U S A. 2005, 102: 15545-15550. 10.1073/pnas.0506580102.PubMed CentralView ArticlePubMedGoogle Scholar
- Segal E, Friedman N, Koller D, Regev A: A module map showing conditional activity of expression modules in cancer. Nature Genetics. 2004, 36 (10): 1090-1098. 10.1038/ng1434.View ArticlePubMedGoogle Scholar
- Härdle W, Müller M, Sperlich S, Werwatz A: Nonparametric and Semiparametric Models. 2004, Springer Series in Statistics, Berlin: SpringerView ArticleGoogle Scholar
- Khan S, Bandyopadhyay S, Ganguly A, Saigal S: Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Physical Review E. 2007, 76: 026209-10.1103/PhysRevE.76.026209.View ArticleGoogle Scholar
- Vaart van der AW, Wellner JA: Weak Convergence and Empirical Processes. With Applications to Statistics. 1996, Springer, New YorkView ArticleGoogle Scholar
- Geer van de S: Empirical Processes in M-Estimation. 2000, Cambridge University PressGoogle Scholar
- Geer van de S: Estimating a Regression Function. The Annals of Statistics. 1990, 18 (2): 907-924. 10.1214/aos/1176347632.View ArticleGoogle Scholar
- Birgé L, Massart P: Rates of convergence for minimum contrast estimators. Probability Theory and Related Fields. 1993, 97: 113-150. 10.1007/BF01199316.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.