Empirical Bayes estimation of posterior probabilities of enrichment: A comparative study of five estimators of the local false discovery rate
 Zhenyu Yang^{1},
 Zuojing Li^{2} and
 David R Bickel^{1}Email author
DOI: 10.1186/147121051487
© Yang et al.; licensee BioMed Central Ltd. 2013
Received: 20 December 2011
Accepted: 11 February 2013
Published: 6 March 2013
Abstract
Background
In investigating differentially expressed genes or other selected features, researchers conduct hypothesis tests to determine which biological categories, such as those of the Gene Ontology (GO), are enriched for the selected features. Multiple comparison procedures (MCPs) are commonly used to prevent excessive false positive rates. Traditional MCPs, e.g., the Bonferroni method, go to the opposite extreme: strictly controlling a familywise error rate, resulting in excessive false negative rates. Researchers generally prefer the more balanced approach of instead controlling the false discovery rate (FDR). However, the qvalues that methods of FDR control assign to biological categories tend to be too low to reliably estimate the probability that a biological category is not enriched for the preselected features. Thus, we study an application of the other estimators of that probability, which is called the local FDR (LFDR).
Results
We considered five LFDR estimators for detecting enriched GO terms: a binomialbased estimator (BBE), a maximum likelihood estimator (MLE), a normalized MLE (NMLE), a histogrambased estimator assuming a theoretical null hypothesis (HBE), and a histogrambased estimator assuming an empirical null hypothesis (HBEEN). Since NMLE depends not only on the data but also on the specified value of Π_{0}, the proportion of nonenriched GO terms, it is only advantageous when either Π_{0} is already known with sufficient accuracy or there are data for only 1 GO term. By contrast, the other estimators work without specifying Π_{0} but require data for at least 2 GO terms. Our simulation studies yielded the following summaries of the relative performance of each of those four estimators. HBE and HBEEN produced larger biases for 2, 4, 8, 32, and 100 GO terms than BBE and MLE. BBE has the lowest bias if Π_{0} is 1 and if the number of GO terms is between 2 and 32. The bias of MLE is no worse than that of BBE for 100 GO terms even when the ideal number of components in its underlying mixture model is unknown, but has high bias when the number of GO terms is small compared to the number of estimated parameters. For unknown values of Π_{0}, BBE has the lowest bias for a small number of GO terms (232 GO terms), and MLE has the lowest bias for a medium number of GO terms (100 GO terms).
Conclusions
For enrichment detection, we recommend estimating the LFDR by MLE given at least a medium number of GO terms, by BBE given a small number of GO terms, and by NMLE given either only 1 GO term or precise knowledge of Π_{0}.
Background
The development of microarray techniques and highthroughput genomic, proteomic, and bioinformatics scanning approaches (such as microarray gene expression profiling, mass spectrometry, and ChIPonchip) has enabled researchers to simultaneously study tens of thousands of biological features (e.g., genes, proteins, singlenucleotide polymorphisms [SNPs], etc.), and to identify a set of features for further investigation. However, there remains the challenge of interpreting these features biologically. For a given set of features, the determination of whether some biological information terms are enriched (i.e., differentially represented), compared to the reference feature set, is termed the feature enrichment problem. The biological information term may be, for instance, a Gene Ontology (GO) term [1, 2] or a pathway in the Kyoto Encyclopedia of Genes and Genomes (KEGG) [3]. We call this problem the feature enrichment problem.
This problem has been addressed using a number of highthroughput enrichment tools, including DAVID, MAPPFinder, OntoExpress and GoMiner [47]. Huang et al. [8] reviewed 68 distinct feature enrichment analysis tools. These authors further classified feature enrichment analysis tools into 3 categories: singular enrichment analysis (SEA), gene set enrichment analysis (GSEA), and modular enrichment analysis (MEA). In this article, we propose empirical Bayes solutions to the SEA problem using genes as archetypal features. Without loss of generality, we consider whether some specific biological categories are enriched for differentially expressed genes with respect to the reference genes.
Indeed, like other enrichmentdetection methods, our methods apply much more broadly. They can assess enrichment given any sublist of features selected for future study, not just a list of genes considered differentially expressed. An anonymous referee pointed out these examples of such lists of candidate features that arise in the context of whole genome sequencing:

genes with SNPs

genes with copy number variations

genes with loss of heterozygosity
These examples and those of our first paragraph do not exhaust contemporary applications, and the feature enrichment problem may occur in unforeseen domains of study. Thus, our illustrative use of differential gene expression as a running example should not be interpreted as a limitation.
Here and in the remainder of the paper, we use GO terms as concrete examples of biological categories without excluding applications of the methods to categories from other relevant databases. The general process begins as follows:

For each GO term, construct Table 1 based on the preselected genes (e.g., differentially expressed (DE) genes) and reference genes (e.g., all genes measured in a microarray experiment).

Compute the pvalue for each GO term using a statistical test that can detect enrichment for the preselected genes.
The number of differentially expressed (DE) and equivalently expressed (EE) genes in a GO category
DE genes  EE gene  Total  

In GO category  x _{1}  x _{2}  x_{1}+x_{2} 
Not in GO category  n − x_{1}  N − n − x_{2}  N − x_{1} − x_{2} 
Total  n  N − n  N 
Multiple comparison procedures (MCPs) are then applied to the resulting pvalues to prevent excessive false positive rates. The false discovery rate (FDR) [9] is frequently used to control the expected proportion of incorrectly rejected null hypotheses in gene enrichment studies [1012] because it has lower false negative rates than Bonferroni correction and other methods of controlling the familywise error rate. Methods of FDR control assign qvalues [13] to biological categories, but qvalues are too low to reliably estimate the probability that the biological category is not enriched for the preselected features. Thus, we study application of better estimators of that probability, which is technically known as the local FDR (LFDR). Hong et al. [14] used an LFDR estimator to solve a GSEA problem and pointed out that this was less biased than the qvalue for estimating the LFDR, the posterior probability that the null hypothesis is true.
Efron [15, 16] devised reliable LFDR estimators for a range of applications in microarray gene expression analysis and other problems of largescale inference. However, whereas microarray gene expression analysis takes into account tens of thousands of genes, the feature enrichment problem typically concerns a much smaller number of GO terms. While these methods are appropriate for microarrayscale inference, they are less reliable for enrichmentscale inference [1719]. Thus, we will specifically adapt LFDR estimators that are appropriate for smallerscale inference to address the SEA problem. Again, we will focus on genes and GO terms for the sake of concreteness. Nevertheless, the estimators used can be applied to other features and to other biological terms (e.g., metabolic pathways).
The sections of this paper are arranged as follows. We first introduce some preliminary concepts in the feature enrichment problem. Next, two previous LFDR estimators and three new LFDR estimators are described. Following this, we compare the LFDR estimators by means of a simulation study and an application to breast cancer data. Finally, we draw conclusions and make recommendations on the basis of our results.
Preliminary concepts
The feature enrichment problem described in the Background section is stated here more formally for the application of LFDR methods in the next section.
Likelihood functions
In Table 1, x_{1} and x_{2} are the observed numbers DE genes and EE genes in a given GO category, respectively. Whereas n is the total number of DE genes, N is the total number of reference genes. Thus, N − n is the total number of EE genes. The columns gives the numbers of DE genes and EE genes, and the rows give the numbers of genes in the GO category and outside the GO category.
where 0 ≤ x_{1} ≤ n, 0 ≤ x_{2} ≤ N − n, and 0 ≤ π_{ i } ≤ 1, i=1,2.
where 0 ≤ x_{1} ≤ n and 0 ≤ x_{2} ≤ N − n.
understood as a function of t.
By eliminating the nuisance parameter λ, we can reduce the original data x_{1} and x_{2} by considering the statistic T = t. However, the use of the conditional probability mass function requires some justification because of concerns about losing information during the conditioning process. Unfortunately, in the presence of the nuisance parameter, the statistic S(X_{1},X_{2}) = X_{1}+X_{2} is not an ancillary statistic for the parameter of interest. In other words, the probability mass function of the conditional variable S(X_{1},X_{2}) may contain some information about parameter θ[20]. However, following the explanation of BarndorNielsen and Cox ([21], $\S 2.5$), the expectation value of statistic S(X_{1},X_{2}) equals the nuisance parameter. Hence, from the observation of S(X_{1},X_{2}) alone, the distribution of S(X_{1},X_{2}) contains little information about θ[21]. S(X_{1},X_{2}) satisfies the other 3 conditions of an ancillary statistic defined by BarndorNielsen and Cox [21]: parameters θ and λ are variation independent; (T(X_{1},X_{2}),S(X_{1},X_{2})) is the minimal sufficient statistic; and the distribution of T(X_{1},X_{2}), given S(X_{1},X_{2}) = s, is independent of the parameter of interest, θ, given the nuisance parameter λ. Therefore, the probability mass function of S(X_{1},X_{2}) contains little information about the value of θ.
Hypotheses and LFDRs
where ω_{ i } is defined in equation (9).
Methods
 1.
Previous LFDR estimators. While not unique to this paper, these methods are included for comparison.
 2.
New LFDR estimators. Our main methodological innovations are the uses of a conditional probability mass function and of normalized maximum likelihood for LFDR estimation.
The other original contributions of this paper are the estimator comparisons of the next section. The comparisons are made by simulation and by a case study.
Previous LFDR estimators
Binomialbased LFDR estimator
The version of the FDR that generalizes the LFDR is the nonlocal FDR, which is defined as the ratio of the expected number of false discoveries to the expected total number of discoveries [17]. In our running example, a discovery of enrichment is a rejection of the null hypothesis of nonenrichment at some significance level α, and a false discovery of enrichment is a discovery of enrichment corresponding to a case of no actual enrichment. (This FDR has been called the “Bayesian FDR” [22] to distinguish it from the FDR of Benjamini and Hochberg [9]).
where m is the number of GO terms, p_{ j } is the pvalue of GO term j, and ${\mathbf{1}}_{\{{p}_{j}\le \alpha \}}$ is the indicator such that ${\mathbf{1}}_{\{{p}_{j}\le \alpha \}}=1$ if p_{ j } ≤ α is true and ${\mathbf{1}}_{\{{p}_{j}\le \alpha \}}=0$ otherwise. Thus, $\sum _{j=1}^{m}{\mathbf{1}}_{\{{p}_{j}\le \alpha \}}$ represents the number of discoveries of enriched GO terms, and m α estimates the number of such discoveries that are false.
It is conservative in the sense that it tends to overestimate LFDR [17].
Histogrambased LFDR estimator
where $\widehat{f}$ is the estimator of f that is estimated by a nonparametric Poisson regression method [15, 16]. We call ${\hat{\text{LFDR}}}_{i}$the histogrambased estimator (HBE) if the density function f_{0} is assumed to be standard normal, N(0,1), and the histogrambased estimator with empirical null (HBEEN) if the density function f_{0} is estimated based on the truncated maximum likelihood technique of [16]. Dalmasso et al. [23] compared the precursor of HBEEN [15] to other LFDR estimators.
New LFDR estimators
Type II maximum likelihood estimator
Bickel [17] follows Good [24] in calling the maximization of likelihood over a hyperparameter Type II maximum likelihood to distinguish it from the usual Type I maximum likelihood, which pertains only to models that lack random parameters. Type II maximum likelihood has been applied to parametric mixture models (PMMs) for the analysis of microarray data [25, 26], proteomics data [18], and genetic association data [27]. In this section, we adapt the approach to the feature enrichment problem by using the conditional probability mass function defined above. The particular models we use in this framework correspond to new methods of enrichment analysis.
where θ_{0} = 0 and θ_{ j }≠θ_{ J } for all $j,J\in \left\{0,\dots ,k1\right\}$ such that j≠J.
where S_{ i } is the observed value of S_{ i } for GO term i, and S=〈S_{1},S_{2},⋯,S_{ m }〉.
Moreover, we assume that for a given number of genes in GO term i, T_{ i }(i = 1,…,m) satisfies the kcomponent PMM shown in equation (16). In other words, we assume that the possible log odds ratios of GO term i are the θ_{0},θ_{1},θ_{2},…,θ_{k − 1} of equation (16) if the alternative hypothesis H_{1} in hypothesis comparison (7) is true.
where ${\hat{\theta}}_{1},\dots ,{\hat{\theta}}_{k1}$ and ${\hat{\Pi}}_{0},\dots ,{\hat{\Pi}}_{k1}$ are maximum likelihood estimates of θ_{1},…,θ_{k − 1} and Π_{0},…,Π_{k − 1} in equation (18). We call ${\hat{\text{LFDR}}}_{i}^{\left(k\right)}$ the k component maximum likelihood estimator (MLEk). Our LFDRenrich and LFDRhat software suites of R functions that implement MLE2 and MLE3 are now available at http://www.statomics.com. Moreover, ${\hat{\theta}}_{i}$ (i = 1,…,k − 1;k = 2,3) and ${\hat{\Pi}}_{j}$ (j = 0,…,k − 1;k = 2,3), also in LFDRenrich and LFDRhat, are maximum likelihood estimators of θ_{ i } (i = 1,…,k − 1;k = 2,3) and Π_{ j } (j = 0,…,k − 1;k = 2,3) under given constraints.
LFDR estimator based on the normalized maximum likelihood
Therefore, given a guessed value of Π_{0}, we may use an estimator of the Bayes factor to estimate the LFDR of a GO term.
where ${\widehat{\theta}}_{i}\left({t}_{i}\right{s}_{i})$ is a Type I MLE with respect to $\mathcal{\mathscr{H}}$ under observed values T_{ i } given S_{ i }[28, 29].
We call ${f}_{i}^{\u2020}\left({t}_{i}\right{s}_{i};\mathcal{\mathscr{H}})$ the normalized maximum likelihood (NML) associated with the hypothesis that ${\theta}_{i}\in \mathcal{\mathscr{H}}$.
where ${\hat{\text{BF}}}_{i}^{\u2020}$ is defined in equation (26). We call this LFDR estimator the NML estimator (NMLE).
as the empirical Bayes estimator of BF_{ i }.
Results and discussion
In this section, we compared the LFDR estimators using simulation data and breast cancer data.
For each GO category, the pvalue used in BBE to estimate LFDR is computed based on the 2sided Fisher’s exact test. In computing MLEk (k = 2,3), θ_{ i } (i = 1,…,k − 1) in equation (18) was constrained to lie between 0 and 10, whereas Π_{ i } (i = 0,…,k − 1) in equation (18) was allowed to take any value between 0 and 1 such that $\sum _{i=0}^{k1}{\Pi}_{i}=1$.
Simulation studies
The aim of the following simulation studies is to compare the LFDR estimation biases of BBE, MLE2, MLE3, HBE, and HBEEN. NMLE is not taken into account because its performance depends not only on the data, but also on the specified prior probability Π_{0}.
The simulation setting involves 10,000 genes in a microarray with 200 genes identified as DE and 100 GO terms. We conducted a separate simulation study using each of these values of Π_{0}: 50%, 60%, 70%, 80%, 90%, and 94%.
Considering the log odds ratios of all GO terms in each simulation study, we generated Table 1 for GO term i and for each of the 20 simulated data sets as follows:

x_{1} is generated from a binomial distribution with parameter π_{1} used in equation (2); π_{1} is a real value randomly picked from 0 to 1.

x_{2} is obtained from a binomial distribution with parameter ${\pi}_{2}={\left[\frac{(1{\pi}_{1})\times {2}^{{\varphi}_{i}}}{{\pi}_{1}}+1\right]}^{1}$, obtained by solving${\varphi}_{i}=\underset{2}{log}[{\pi}_{1}/(1{\pi}_{1}\left)\right]\underset{2}{log}[{\pi}_{2}/(1{\pi}_{2}\left)\right].$(31)
Thus, according to equation (4), we obtain ϕ_{ i } = θ_{ i } log2e for GO term i. Each of those artificial data sets represents what might have been a real data set such as that of the next subsection.
where f_{ θ }(ts) is defined in equation (6).
The proportion of nonenriched GO terms and the log2 odds ratios of GO terms used in the simulation studies
Number of GO terms  Π _{0}  log2 odds ratio (ϕ_{ i })  

0.5,  i = 1  
2  50.0%  0,  i = 2 
100.0%  0,  i = 1,2  
3,  i = 1  
50.0%  −3,  i = 2  
0,  i = 3,4  
0.5,  i = 1  
4  75.0%  0,  i = 2,…,4 
100.0%  0,  i = 1,…,4  
(0.5 +1.5×(i − 1)),  i=1,2  
50.0%  −(0.5 +1.5×(i − 3)),  i = 3,4  
0,  i = 5,…,8  
(0.5 +1.5×(i − 1)),  i = 1,2  
62.5%  −(0.5 +1.5×(i − 2)),  i = 3  
0,  i = 4,…,8  
2,  i = 1  
8  75.0%  −2,  i = 2 
0,  i=3,…,8  
2,  i = 1  
87.5%  0,  i = 2,…,8  
100.0%  0,  i = 1,…,8  
(0.32 +0.64×(i − 1)),  i = 1,…,8  
50.0%  −(0.32 +0.64×(i − 9)),  i = 9,…,16  
0,  i = 17,…,32  
(0.8 +0.8×(i − 1)),  i = 1,…,3  
62.5%  −(0.8 +0.8×(i − 4)),  i = 4,…,6  
0,  i = 7,…,32  
(0.2 +1.6×(i − 1)),  i = 1,2  
32  75.0%  −(0.2 +1.6×(i − 1)),  i = 3,4 
0,  i = 5,…,32  
1.8,  i = 1  
87.5%  −1.8,  i = 2  
0,  i = 3,…,32  
100.0%  0,  i = 1,…,32 
Breast cancer data analysis
The singlechannel microarray data set used here to illustrate our new methods is from an experiment applying an estrogen treatment to cells of a human breast cancer cell line [31]. The Affymetrix human genome U95Av2 genechip data are from four samples from an estrogen receptor positive breast cancer cell line. Two of the samples were exposed to estrogen and then harvested after 10 hours. The remaining two samples were left untreated and then harvested after 10 hours. For simplicity of terminology, we call probes in the microarray experiment “genes.” The relevant data consist of measurements of gene expression across the reference class of 12,625 genes. The purpose of the study was to determine which genes are affected by the estrogen treatment. (For further information concerning the data, see Gentleman et al. [32].)
We applied the R function expresso in the affy package [33] of Bioconductor [34] to convert the raw probe intensities from the the CEL data files to logarithms of gene expression levels without background correction. In doing so, we applied the “quantiles,” “pmonly,” and “medianpolish” [35] preprocessing settings.
We selected as genes of interest those that were differentially expressed between the treatment group and the control group according to the following criterion. Using the LFDR as the probability that a gene is EE, we considered genes with LFDR estimates below 0.2 as DE. In other words, we selected as DE genes those that were differentially expressed with estimated posterior probability of at least 80%. Considering four samples of each gene in the microarray, we used the unpaired ttest with equal variances to compute the pvalue. The LFDR of every gene is estimated using the theoretical null hypothesis method of Efron [15, 16]; the empirical null hypotheses method can lead to excessive bias due to deviations from normality [36]. When we compared gene expression data for the presence and absence of estrogen after 10 hours of exposure, we obtained 74 DE genes.
For two GO terms, opposite conclusions would be drawn about their enrichment, depending on which estimator is used. As seen in Figure 4, the estimated LFDRs of GO:0051082 and GO:0005524 using MLE2 were 100%. However, the LFDRs estimated by MLE3 were essentially 0.
Using the MLE formula shown in equation (19), and the kcomponent PMM shown in equation (16), we conclude that the sensitivity of the LFDRs of GO term i estimated by MLE2 and MLE3 depended mainly on the sensitivity of the Bayes factor, based on the number of PMM components. Comparing the probability masses of GO:0005524, based on the 2 and 3component PMMs shown in Figure 5, we found that the probability mass of GO:0005524 under the null hypothesis is larger than that under the alternative hypothesis based on the 2component PMM (left plot in Figure 5). In contrast, the probability mass under the null hypothesis is smaller than that under the alternative hypothesis based on the 3component PMM (right plot in Figure 5). Thus, the LFDR estimated by MLE is strongly dependent on the number of PMM components.
While a real data set can in that way indicate the impact of selecting an appropriate method, that impact does not in itself say which method has lowest bias. For that, we rely on the simulation study of the previous subsection.
Conclusions
As seen in Figure 1 and Figure 2, HBE and HBEEN have relative high biases for a small number and a medium number of GO terms, respectively. The performance comparison displayed in the lefthand side of Figure 3 indicates that BBE contains the lowest minimum estimated LFDR bias for a small number of GO terms (i.e., 232 GO terms) when the proportion of nonenriched GO terms is 1. Although the minimum bias of BEE is not the lowest for some Π_{0}s under a small number of GO terms, it is very close to the lowest value of bias based on plots shown in Figure 2. The righthand side of Figure 3 indicates that MLE3 has the lowest maximum absolute estimated LFDR bias in 100 GO terms. MLE exhibits bias similar to that of BBE when the number of GO terms is much larger than k except for when the proportion of nonenriched GO terms is high (close to 1). Moreover, MLE3 has lower bias than MLE2 as an LFDR estimator. Due to its conservatism and freedom from PMM, we recommend using BBE for a small number of GO terms of interest (232 GO terms) and MLE for a medium number of GO terms of interest (100 GO terms).
Finally, we recommend that NMLE be used when there is only 1 GO term of interest since none of the other estimators is able to estimate LFDR in such a case except by conservatively giving 1 as the estimate. Otherwise, unless Π_{0} is known with sufficient accuracy, NMLE is not recommended since it depends not only on the data but also on a guess of the value of Π_{0}, which in the absence of strong prior information, is often set to the default value of 50%. A closely related approach is to use the logarithm of the NML ratio as a measure of statistical support for the enrichment hypothesis [30] without guessing Π_{0}. By using 10 and 100 as thresholds of the approximate Bayes factors from equations (26) and (28) to determine whether a GO term is enriched, we reached similar conclusions with both NML and MLE (Figure 6). Thus, in our data set, the NML ratio tends to estimate the Bayes factor almost as accurately as methods that simultaneously use information across GO terms. While we do not expect the same for all data sets, we note that similar results have been found for an application of a modified NML [29] to a proteomics data set [30].
Declarations
Acknowledgements
We thank the two anonymous reviewers for comments that led to improvements in the manuscript, particularly those that led to clearer communication or to additional simulation studies. We also thank both Editage and Donna Reeder for the detailed copyediting. We are grateful to Corey Yanofsky and Ye Yang for the useful discussions. This work was partially supported by the Natural Sciences and Engineering Research Council of Canada, by the Canada Foundation for Innovation, by the Ministry of Research and Innovation of Ontario, and by the Faculty of Medicine of the University of Ottawa.
Authors’ Affiliations
References
 Altshuler D, Daly MJ, Lander ES: Genetic mapping in human disease. Science 2008, 322: 881888. 10.1126/science.1156409PubMed CentralView ArticlePubMedGoogle Scholar
 Rhee SY, Wood V, Dolinski K, Draghici S: Use and misuse of the gene ontology annotations. Nat Rev Genet 2008,9(7):509515. 10.1038/nrg2363View ArticlePubMedGoogle Scholar
 Kanehisa M, Goto S: KEGG: Kyoto Encyclopedia of Genes and Genome. Nucleic Acids Res 2000, 28: 2730. 10.1093/nar/28.1.27PubMed CentralView ArticlePubMedGoogle Scholar
 Dennis G, Sherman BT, Hosack DA, Yang J, Gao W, Lane HC, Lempicki R: DAVID: Database for Annotation, Visualization, and Integrated Discovery. Genome Biol 2003, 4: P3. 10.1186/gb200345p3View ArticlePubMedGoogle Scholar
 Doniger SW, Salomonis N, Dahlquist KD, Vranizan K, Lawlor SC, Conklin BR: MAPPFinder: using gene ontology and GenMAPP to create a global geneexpression profile from microarray data. Genome Biol 2003, 4: R7. 10.1186/gb200341r7PubMed CentralView ArticlePubMedGoogle Scholar
 Khatri P, Draghici S, Ostermeier G, Krawetz S: Profiling gene expression using ontoexpress. Genomics 2002, 79: 266270. 10.1006/geno.2002.6698View ArticlePubMedGoogle Scholar
 Zeeberg BR, Feng W, Wang G, Wang MD, Fojo AT, Sunshine M, Narasimhan S, Kane DW, Reinhold WC, Lababidi S, Bussey KJ, Riss J, Barrett JC, Weinstein JN: GoMiner: a resource for biological interpretation of genomic and proteomic data. Genome Biol 2003, 4: R28. 10.1186/gb200344r28PubMed CentralView ArticlePubMedGoogle Scholar
 Huang DW, Sherman BT, Lempicki RA: Bioinformatics enrichment tools paths toward the comprehensive functional analysis of large gene lists. Nucleic Acids Res 2009, 37: 113. 10.1093/nar/gkn923PubMed CentralView ArticleGoogle Scholar
 Benjamini Y, Hochberg Y: Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B 1995, 57: 289300.Google Scholar
 Min JL, Barrett A, Watts T, Pettersson FH, Lockstone HE, Lindgren CM, Taylor JM, Allen M, Zondervan KT, McCarthy MI: Variability of gene expression profiles in human blood and lymphoblastoid cell lines. BMC Genomics 2010, 11: 96. 10.1186/147121641196PubMed CentralView ArticlePubMedGoogle Scholar
 Reyal F, van Vliet MH, Armstrong NJ, Horlings HM, de Visser KE, Kok M, Teschendorff AE, Mook S, van’t Veer L, Caldas C, Salmon RJ, Vijver MJVD, Wessels LFA: A comprehensive analysis of prognostic signatures reveals the high predictive capacity of the proliferation, immune response and RNA splicing modules in breast cancer. Breast Cancer Res 2008, 10: R93. 10.1186/bcr2192PubMed CentralView ArticlePubMedGoogle Scholar
 Wang R, Bencic D, Lazorchak J, Villeneuve D, Ankley GT: Transcriptional regulatory dynamics of the hypothalamicpituitarygonadal axis and its peripheral pathways as impacted by the 3beta HSD inhibitor trilostane in zebrafish (Danio rerio). Ecotoxicol Environ Saf 2011, 74: 14611470. 10.1016/j.ecoenv.2011.05.001View ArticlePubMedGoogle Scholar
 Storey JD: The positive false discovery rate: a Bayesian interpretation and the qvalue. Ann Stat 2003, 31: 20132035. 10.1214/aos/1074290335View ArticleGoogle Scholar
 Hong WJ, Tibshirani R, Chu G: Local false discovery rate facilitates comparison of different microarray experiments. Nucleic Acids Res 2009, 37: 74837497. 10.1093/nar/gkp813PubMed CentralView ArticlePubMedGoogle Scholar
 Efron B: Largescale simultaneous hypothesis testing: The choice of a null hypothesis. J Am Stat Assoc 2004, 99: 96104. 10.1198/016214504000000089View ArticleGoogle Scholar
 Efron B: LargeScale Inference: Empirical Bayes Methods for Estimation,Testing, and Prediction Cambridge. Cambridge University Press; 2010.View ArticleGoogle Scholar
 Bickel DR: Simple estimators of false discovery rates given as few as one or two pvalues without strong parametric assumptions. Stat Appl Genet Mol Biol in press in pressGoogle Scholar
 Bickel DR: Smallscale inference: empirical Bayes and confidence methods for as few as a single comparison. Tech Rep, Ottawa Inst Syst Biol; 2011:arXiv:1104.0341arXiv:1104.0341.Google Scholar
 Padilla M, Bickel DR: Empirical Bayes methods corrected for small numbers of tests. Stat Appl Genet Mol Biol 2012,11(5):art. 4.Google Scholar
 Severini T: Likelihood Methods in Statistics Oxford. Oxford University Press; 2000.Google Scholar
 BarndorffNielsen OE, Cox DR: Inference and Asymptotics. London: CRC Press; 1994.View ArticleGoogle Scholar
 Efron B, Tibshirani R: Empirical Bayes methods and false discovery rates for microarrays. Genet Epidemiol 2002, 23: 7086. 10.1002/gepi.1124View ArticlePubMedGoogle Scholar
 Dalmasso C, BarHen A, Broët P: A constrained polynomial regression procedure for estimating the local false discovery rate. BMC Bioinformatics 2007, 8: 229. 10.1186/147121058229PubMed CentralView ArticlePubMedGoogle Scholar
 Good IJ: How to estimate probabilities. IMA J Appl Math 1966, 2: 364383. 10.1093/imamat/2.4.364View ArticleGoogle Scholar
 Pawitan Y, Murthy K, Michiels S, Ploner A: Bias in the estimation of false discovery rate in microarray studies. Bioinformatics 2005, 21: 38653872. 10.1093/bioinformatics/bti626View ArticlePubMedGoogle Scholar
 Muralidharan O: An empirical Bayes mixture method for effect size and false discovery rate estimation. Ann Appl Stat 2010, 4: 422438.View ArticleGoogle Scholar
 Yang Y, Aghababazadeh FA, Bickel DR: Parametric estimation of the local false discovery rate for identifying genetic associations. IEEE/ACM Trans Comput Biol Bioinformatics 2012. online ahead of print at http://dx.doi.org/10.1109/TCBB.2012.140 online ahead of print atGoogle Scholar
 Grünwald PD: The Minimum Description Length Principle. London: MIT Press; 2007.Google Scholar
 Bickel DR: A predictive approach to measuring the strength of statistical evidence for single and multiple comparisons. Can J Stat 2011, 39: 610631. 10.1002/cjs.10109View ArticleGoogle Scholar
 Bickel DR: Minimaxoptimal strength of statistical evidence for a composite alternative hypothesis. Int Stat Rev 2013. in press. 2011 version available at arXiv:1101.0305 in press. 2011 version available at arXiv:1101.0305Google Scholar
 Scholtens D, Miron A, Merchant FM, Miller A, Miron PL, Iglehart JD, Gentleman R: Analyzing factorial designed microarray experiments. J Multivariate Anal 2004, 90: 1943. 10.1016/j.jmva.2004.02.004View ArticleGoogle Scholar
 Gentleman R, Carey V, Huber W, Irizarry R, Dudoit S (Eds): Bioinformatics and Computational Biology Solutions Using R and Bioconductor. New York: Springer; 2005.View ArticleGoogle Scholar
 Gautier L, Cope L, Bolstad BM, Irizarry RA: Affy—analysis of Affymetrix Gene Chip data at the probe level. Bioinformatics 2004,20(3):307315. 10.1093/bioinformatics/btg405View ArticlePubMedGoogle Scholar
 Gentleman RC, Carey VJ, Bates DM, Bolstad B, Dettling M, Dudoit S, Ellis B, Gautier L, Ge Y, Gentry J, Hornik K, Hothorn T, Huber W, Iacus S, Irizarry R, Leisch F, Li C, Maechler M, Rossini AJ, Sawitzki G, Smith C, Smyth G, Tierney L, Yang JYH, Zhang J: Bioconductor: open software development for computational biology and bioinformatics. Genome Biol 2004, 5: R80. 10.1186/gb2004510r80PubMed CentralView ArticlePubMedGoogle Scholar
 Tukey JW: Exploratory Data Analysis. Reading: AddisonWesley; 1977.Google Scholar
 Bickel DR: Estimating the null distribution to adjust observed confidence levels for genomescale screening. Biometrics 2011, 67: 363370. 10.1111/j.15410420.2010.01491.xView ArticlePubMedGoogle Scholar
 Jeffreys H: Theory of Probability. London: Oxford University Press; 1948.Google Scholar
 Bickel DR: The strength of statistical evidence for composite hypotheses: inference to the best explanation. Statistica Sinica 2012, 22: 11471198.Google Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.