 Methodology article
 Open Access
A unified approach to false discovery rate estimation
 Korbinian Strimmer^{1}Email author
https://doi.org/10.1186/147121059303
© Strimmer; licensee BioMed Central Ltd. 2008
Received: 19 February 2008
Accepted: 09 July 2008
Published: 09 July 2008
Abstract
Background
False discovery rate (FDR) methods play an important role in analyzing highdimensional data. There are two types of FDR, tail areabased FDR and local FDR, as well as numerous statistical algorithms for estimating or controlling FDR. These differ in terms of underlying test statistics and procedures employed for statistical learning.
Results
A unifying algorithm for simultaneous estimation of both local FDR and tail areabased FDR is presented that can be applied to a diverse range of test statistics, including pvalues, correlations, z and tscores. This approach is semipararametric and is based on a modified Grenander density estimator. For test statistics other than pvalues it allows for empirical null modeling, so that dependencies among tests can be taken into account. The inference of the underlying model employs truncated maximumlikelihood estimation, with the cutoff point chosen according to the false nondiscovery rate.
Conclusion
The proposed procedure generalizes a number of more specialized algorithms and thus offers a common framework for FDR estimation consistent across test statistics and types of FDR. In comparative study the unified approach performs on par with the best competing yet more specialized alternatives. The algorithm is implemented in R in the "fdrtool" package, available under the GNU GPL from http://strimmerlab.org/software/fdrtool/ and from the R package archive CRAN.
Keywords
 False Discovery Rate
 False Discovery Rate Estimation
 Local False Discovery Rate
 False Discovery Rate Procedure
 Observe Test Statistic
Background
The false discovery rate (FDR) plays a prominent role in many highdimensional testing and model selection procedures. Consequently, FDR methodologies are ubiquitous in the analysis of highthroughput data, such as in differential gene expression, SNP biomarker selection, peak detection in proteomic mass spectrometry data, or inference of edges in a network.
False discovery rate analysis starts with the seminal works by Schweder and Spjøtvoll [1] and by Benjamini and Hochberg [2]. Many others have followed suite, so that to date an impressive number of different algorithms for controlling and estimating false discovery rates have appeared in the literature.
In a nutshell, FDR estimation algorithms may be broadly categorized by the type of

FDR,

input test statistic, and

employed inference procedures.
There are two main types of FDR, the "classic" tail areabased FDR (= Fdr) and local FDR (= fdr). Most FDR procedures are concerned either with Fdr or fdr, simultaneous estimation of both types of FDR is only possible with a few selected algorithms. With regard to test statistics, FDR calculations typically rely on pvalues. However, FDR can be easily extended to other test statistics, such as correlations [3]. Relaxing the requirement of having pvalues as input has the additional benefit that it allows for empirical null modeling [4]. Further key differences among the various FDR methods relate primarily to their respective procedures for density estimation and for inferring the proportion of null statistics.
Here, a unified statistical procedure for FDR estimation is described that generalizes a number of previous algorithms, specifically those of [5, 6, 4] and [7]. Notable features of thus approach include simultaneous estimation of Fdr and fdr from a diverse range of test statistics, its simplicity, very little a prior modeling assumptions, and the option of fitting the empirical null model.
The remainder of this paper is set out as follows. In the first part of the 'Methods' section a brief overview is given of the basic theory and definitions related to FDR and its estimation. In the second part of the 'Methods' section the proposed unified FDR procedure is described in detail. In the remaining part of the paper the new procedure is evaluated in comparison with other competing FDR estimation schemes.
Methods
Basic theory of FDR
This section gives a very brief review of the two component FDR model and the local and tail areabased FDR criteria. For a more refined discussion it is referred to [8] and references therein.
Throughout the paper the Efron naming conventions are followed. Specifically, "fdr" denotes the local false discovery rate, "Fdr" denotes the tail areabased false discovery rate, and "FDR" is a generic term encompassing both variants. Similarly, FNDR is the generic abbreviation for the false nondiscovery rate [9].
In the following m simultaneous tests are considered, resulting in m test statistics such as t_{1},...,t_{ m }or z_{1},...,z_{ m }and corresponding pvalues p_{1},...,p_{ m }.
Tail areabased FDR
In order to control the number of false discoveries, i.e. the expected ratio E(V/R) of the number of false positives V among all significant tests R, Benjamini and Hochberg [2] introduced the following linear stepup procedure. First, the pvalues are ordered so that p_{(1)} ≤ ... ≤ p_{(m)}. Second, each value p_{(i)}is compared with $q\frac{i}{m}$, where q is the desired FDR level. Finally, with k = max(i : p_{(i)}≤ $q\frac{i}{m}$) all hypotheses belonging to p_{(1)},...,p_{(k)}are rejected. [2] show that when the test statistics are independent then this procedure controls E(V/R) at level ≤ q.
Here order(p_{ i }) equals one for the smallest and m for the largest pvalue, respectively. For comparison, the standard Bonferroni correction [10] is ${p}_{i}^{\text{Bf}}={p}_{i}m$, and hence ${p}_{i}\le {p}_{i}^{\text{BH}}\le {p}_{i}^{\text{Bf}}$.
Definitions of FDR quantities contrasted with that of specificity and sensitivity.
Quantity  Definition  

Fdr  = Prob("not interesting"Y ≥ y)  $=\frac{FP}{TP+FP}$ 
Fndr  = Prob("interesting"Y ≤ y)  $=\frac{FN}{FN+TN}$ 
Sensitivity, Power  = Prob(Y ≥ y"interesting")  $=\frac{TP}{TP+FN}$ 
Specificity  = Prob(Y ≤ y"not interesting")  $=\frac{TN}{TN+FP}$ 
fdr  = Prob("not interesting"Y = y)  
fndr  = Prob("interesting" Y = y)  = 1  fdr 
It is instructive to compare the definitions of "Fdr" and "Fndr" for a given threshold y with those of "sensitivity" and "specificity" – see Table 1. Note that the order of conditioning is reversed in the two instances, but otherwise the definitions are very similar. Furthermore, both "FdrFndr" and "sensitivityspecificity" offer the means for determining an optimal decision rule. In a conventional test situation the threshold y is chosen to maximize both sensitivity and specificity (i.e. typically specificity is fixed and power is maximized). Analogously, in an FDR analysis one seeks to minimize Fdr and Fndr (e.g, by fixing Fndr and minimizing Fdr). Hence, there is a tradeoff between Fndr and Fdr, just as there is a tradeoff between sensitivity and specificity. Note that the formal similarities between Fdr/Fndr and sensitivity/specificity is yet another reason for prefering the Bayesian variant of FDR over other more operational definitions.
The BH rule is popular due to its simplicity. However, often it is a rather conservative estimator of Fdr. One way to improve the BH rule is to substitute a more appropriate estimate of the null proportion η_{0}. This leads directly to the wellknown qvalues, which are refined BH estimates with various suggested options for the estimation of η_{0} [12, 7].
Additionally, monotonicity is another issue where the BH rule is open to improvement. Specifically, direct application of the BH correction easily yields corrected pvalues with a different ordering than that of the original test statistics. This unpleasant property has already been noted by [2], and indeed the "max" function in the original stepup procedure provides a corresponding fix (albeit a rather adhoc one). [5, 6] point out that this issue can be more elegantly resolved by requiring the distribution function F(p) of the pvalues to be concave and, correspondingly, the marginal density f(p) to be monotonically decreasing. There many different ways for fitting the two component FDR mixture model (Eqs. 2 and 3) and for estimating densities and η_{0}. This explains the multitude of FDR approaches in existence. Common to all is some form of "zero assumption" to render the mixture model identifiable. Typically, for large pvalues one assumes that there is no contamination with the alternative distribution, i.e. Fndr(p → 1) = 0 and therefore f(p → 1) = η_{0}
Local FDR
An alternative to the classic tailarea based FDR is the local FDR, abbreviated here as "fdr". Specifically, the local FDR is the probability of the null model conditioned the observed test statistic (see Table 1). Note that the local FDR takes is computed at the density level, in contrast to the Fdr that is based on cumulative densities.
This approach has mainly been advocated by Efron and a few others [13–15]. The key virtue of local FDR is that it is more readily interpretable than Fdr, as it is an empirical Bayesian posterior probability and not some variant of a corrected pvalue. However, due to the use of densities it is also more difficult to estimate, in particular if the alternative distribution in the twocomponent model is not parametrically specified.
An important relation between Fdr and fdr is the property Fdr(p) ≤ fdr(p) that holds if fdr(p) is monotonically decreasing with decreasing pvalue.
Test statistics other than pvalues and empirical null modeling
Virtually all FDR procedures – both local and tail areabased methods – are designed to work with pvalues as input test statistics. Regardless the popularity of pvaluebased approaches, in many instances it is often more beneficial to base the FDR calculations on the actual test statistic, such as on a regularized tscore, a zscore, or a correlation, rather than on a pvalue.
The reason for this is as follows. Very often the theoretical null model is misspecified, due to dependencies among test statistics and other factors [16]. In turn, this may lead to overly pessimistic or too optimistic pvalues, and thus to a violation of the implicit assumption of the FDR twocomponent model for pvalues (namely that the null pvalues are drawn from the uniform distribution). In such a case the resulting FDR values will also be biased, and thus unreliable.
Efron has shown that this can be elegantly avoided by retaining free parameters in the null model for the original test statistics (typically for location or scale) and estimating these parameters from the data [4]. Intriguingly, this empirical null modeling is greatly facilitated by high dimensions – and hence it is one of the few instances where highdimensionality is not a curse but a blessing. There are various attempts to take account of the dependencies among pvalues in FDR calculations, however it seems much more natural (and easier) to simply conduct Fdr and fdr calculations on the level of the original test statistic whilst employing an empirical null. In a recent paper these considerations are confirmed from a decision theoretic point of view [17].
Despite these apparent advantages empirical null modeling is currently available in only two FDR estimation algorithms, "locfdr" [4] and "fdrtool" (this paper). Note that fitting an empirical null is not tied to zscores and the assumption of a normal null distribution, it is equally well feasible for any other test statistic, e.g., correlations [3].
Unified procedure for FDR estimation
Overview and motivation
Overview over some commonly used FDR estimation procedures.
Name  FDR Type  Input Data  Comments  Description of Algorithm 

fdrtool  fdr, Fdr  pvalues,  Modified Grenander density,  This paper 
zscores,  estimate, empirical null model,  
tscores, correlations.  selection of truncation point by FNDR.  
BUM  fdr, Fdr  pvalues  Completely parametric model.  [27] 
SAGx  fdr, Fdr  pvalues  Grenander density estimate.  
qvalue  Fdr  pvalues  Diverse estimates of η_{0} available.  
nFDR  Fdr  pvalues  Bernstein polynomial density.  [24] 
multtest  Fdr  pvalues  BenjaminiHochberg algorithm.  [2] 
LBE  Fdr  pvalues  Locationbased estimator.  [32] 
locfdr  fdr  zscores  Regression spline density estimate, empirical null model.  
nomi  fdr  zvalues  Normal mixture modeling of nonnull density.  [22] 
LocalFDR  fdr  pvalues  Uses loess smoothing.  [15] 
kerfdr  fdr  pvalues  Kernel density estimator.  [23] 
twilight  fdr  pvalues  KS fit of truncation point.  [30] 
localFDR  fdr  pvalues  Based on stochastic order model.  [33] 
The aim of this paper is to introduce an FDR estimation procedure that brings together many aspects that otherwise are only considered separately into one common and coherent setting. Thus, in a sense this offers a unified algorithm for FDR analysis.
Specifically, a procedure is proposed

for the simultaneous estimation of both Fdr and fdr, regardless of the type of test statistic,

that does not treat pvalues any different from other test statistics,

that maintains the ordering of original test statistics,

that uses efficient and well tested techniques for estimating η_{0} and null distribution,

and that remains (largely) compatible with the well established "locfdr" and "qvalue" algorithms.
Furthermore, the algorithm is conceptually simple. Components in this scheme for Fdr/fdr analysis are a generalized definition of the test statistic, a nonparametric density estimator, an approach of fitting the null model, combined together in an effective fashion.
The present approach, discussed in detail in the following subsections, is best described as a marriage of the nonparametric Grenander approach of [5] and [6] with the empirical null modeling of [4]. An implementation is available in the R package "fdrtool" [19].
Generalized test statistic
Central to the algorithm is a generic definition of the underlying test statistic. Specifically, a statistic y ≥ 0 is considered with properties such that large values of y indicate an "interesting" case, and, conversely, small values close to zero an "uninteresting" case. Examples for suitable statistics y include:

y = 1  p where p is a pvalue,

y = z where z is a normal score,

y = r where r is a correlation, and

y = t where t is a tscore.
Note that the choice of test statistic y automatically implies a corresponding null model f_{0}(y; θ), e.g., the uniform, halfnormal, etc., which possibly contains some parameters θ. In terms of y the twocomponent model becomesf(y) = η_{0}f_{0}(y; θ) + (1  η_{0})f_{ A }(y)
andF(y) = η_{0}F_{0}(y; θ) + (1  η_{0})F_{ A }(y).
Furthermore, the pvalue corresponding to the test statistic y equals 1  F_{0}(y; θ).
Density estimation using a modified Grenander approach
A central part of FDR analysis consists of the estimation of the marginal density f(p) and the associated distribution function F(p) from pvalues p_{ i }corresponding to the observed test statistics y_{ i }.
The most simple approach is to use the empirical cumulative density function (ECDF) as estimator of F(p). Note that the ECDF is the nonparametric maximumlikelihood estimate (NPMLE). The ECDF is very widely used in FDR analysis, including the two most popular FDR approaches (BH rule, "qvalue" algorithm). However, the ECDF has the disadvantage that it requires careful postprocessing in order to achive monotone FDR values. Furthermore, it is a nontrivial issue to derive a density from the ECDF (see, e.g., [15] for an approach using loess smoothing). This important if computation of local FDR values is desired.
Another popular option, pursued in the "locfdr" program, is to estimate the density by spline Poisson regression on the histogram counts [21]. This work extremely well in general but can be problematic if the distribution has a strong peak – which is not uncommon, e.g., for pvalues or partial correlations. Furthermore, this approach introduces additional parameters such as the histogram bin width or the degrees of freedom of the spline, which for some data may need diligent adjustment. In addition, as the approach does not place any monotonicity constraints on the density there is no guarantee that the order of the scores is maintained in the corresponding FDR values.
Other possibilities recently proposed for FDR density estimation include, e.g., normal mixtures [22], kernelbased approaches [23] and Bernstein polynomials [24].
An further alternative approach is provided by the Grenander density estimator [25]. In contrast to most other density estimators it has two main benefits which are highly useful in the context of FDR estimation: it explicitly incorporates monotonicity constraints (to preserve ordering of original test statistics) and provides simultaneous estimates of both PDF and CDF (to allow computation of both fdr and Fdr). Nonetheless, it is only slightly more complicated than than the ECDF. For FDR analysis the Grenander estimator has been first suggested by [5] and by [6].
Unfortunately, the standard Grenander estimator exhibits a severe shortcoming when applied the twocomponent FDR model: it leads to inconsistencies with the estimated η_{0}. This problem is extensively discussed in [5], and in fact causes these authors to abandon the Grenander estimator despite its favorable properties.
The point that is made here is that this deficiency can be easily fixed. Specifically, it is argued that the Grenander estimator is indeed very well suited for FDR analysis, but needs further modification in order to satisfy the additional constraints imposed by the twocomponent model.
The key problem can be understood best by going back to Eq. 3 which describes the FDR pvalue mixture model on the level of the CDF. Effectively, this equation implies two constraints that any distribution compatible with the twocomponent model must satisfy:

First, the CDF has to fulfill the condition F(p) ≥ η_{0}p because F(p) = η_{0}p + (1  η_{0})F_{ A }(p).

Second, the condition 1  F(p) ≥ η_{0}(1  p) must be met, because of
1  F(p) = η_{0}(1  p) + (1  η_{0})(1  F_{ A }(p)).
 1.
Compute the ECDF on the basis of the pvalues.
 2.
Given η_{0} impose the mixture model conditions on the ECDF for the pvalues. Specifically, set ${\widehat{F}}^{\prime}$(p_{ i }) = η_{0}p_{ i }if $\widehat{F}$(p_{ i }) <η_{0}p_{ i }(i.e. obey the lower boundary shown in Fig. 3) and likewise set ${\widehat{F}}^{\prime}$(p_{ i }) = 1  η_{0} (1  p_{ i }) if $\widehat{F}$(p_{ i }) > 1  η_{0} (1  p_{ i }) (upper boundary).
 3.
The "modified" Grenander estimator is obtained as the standard Grenander estimator computed from the modified ECDF.
Estimation of null subdensity by truncated maximumlikelihood
For computing pvalues and the modified Grenander density suitable estimates of the parameters θ and η_{0} are required. In other words, the null subdensity η_{0}f_{0}(y; θ) of the twocomponent model (Eqs. 4 and 5) needs to be fit to the observed test statistics. This is straightforward in fully parametric models such as BUM [27]. However, it is often preferred to leave f_{ A }(y) unspecified. As a consequence, standard procedures for inferring mixture models such as the EM algorithm cannot be applied.
Instead, a truncated maximumlikelihood approach is applied here. In more detail, in this method the data are censored at some threshold y_{ c }, so that only test statistics y_{ t }= {y_{ i }: y_{ i }<y_{ c }} are retained. The underlying assumptions is that for y_{ i }<y_{ c }(nearly) all data points belong to the null part. This is called the "strong zero assumption" in [28]. The truncated null density becomes ${f}_{0}^{t}$(y; θ) = f_{0}(y; θ)/F (y_{ c }; θ) for y < y_{ c }and ${f}_{0}^{t}$ = 0 otherwise. Maximization of the corresponding likelihood function returns ${\widehat{\eta}}_{0}$ as well as an estimate of its asymptotic error. Similarly, the proportion of null values η_{0} is inferred by assuming a binomial model for the observed number m_{ t }of hypotheses in the set y_{ t }, which leads to the simple estimate ${\widehat{\eta}}_{0}=\mathrm{max}\{1,\frac{{m}_{t}}{m}/F({y}_{c};\widehat{\theta})\}$ plus an associated error.
Truncated maximumlikelihood is the basis of the "locfdr" MLE algorithm [29, 28]. If the test statistics are pvalues then the truncated maximumlikelihood algorithm reduces to the simple cutoff technique used in "qvalue" and most other pvaluebased packages.
Selection of suitable truncation point using the false nondiscovery rate
Fitting the null model and the associated parameters θ and η_{0} by truncated maximumlikelihood depends on the choice of a suitable cutoff point y_{ c }. In general, one wishes to select y_{ c }such that the threshold is small enough to ensure that the zero assumption is met and that there is minimal bias due to contamination with the alternative f_{ A }(y). On the other hand, y_{ c }needs be chosen large enough so that the number of data points in y_{ t }is sufficient for reliably estimating θ and η_{0}.
The default "smoothing" approach employed in "qvalue" specifically aims at achieving an unbiased estimate of η_{0} [7]. This is obtained by varying the cutoff point between zero and one, and subsequently estimating η_{0} by interpolation at y_{ c }= 1, i.e. for complete censoring.
Various choices of normal truncation points implemented in "locfdr".
Version  Released  Truncation Point  Reference 

1.1–1  July 2006  z_{0} = 2  
1.1–3  December 2006  ${z}_{0}=\widehat{\mu}+b\widehat{\sigma}$ with b = 3.55 – 0.44 log_{10}(m), $\widehat{\mu}$ = median(z_{ i }) and $\widehat{\sigma}$ = IQR(z_{ i })/1.349.  
1.1–6  November 2007  as version 1.1–3, but with b = max(1, 4.3m^{0.112966})  [28] 
A simple approximate fit of the null model is achieved by matching its median ${F}_{0}^{1}$ (1/2; θ) with that of the observed y_{ i }. Thus, a robust estimate of scale is used just as in the "locfdr" algorithm, see Table 3 (note that the median for the halfnormal distribution corresponds to the interquartile range (IQR) of the corresponding normal with mean zero). Subsequently, after converting the test statistics into pvalues an approximate estimate of the null proportion is determined by estimating η_{0} for various cutoffpoints and finally settling for the 0.1 quantile of the resulting distribution of corresponding η_{0}.
In addition to selecting y_{ c }by the above "FNDR" approach, further methods available in the "fdrtool" package include the "locfdr" cutoff method [28] and the specification of the fraction of data points to be considered for estimating the empirical null. In a practical analysis it is always advisable to conduct the FDR calculations for various choices of y_{ c }, (even though truncated maximumlikelihood appears to be fairly robust with regard to y_{ c }).
Gluing it all together
 1.
Determine a suitable truncation point y_{ c }.
 2.
Estimate the null model and its parameters, yielding ${\widehat{\eta}}_{0}$ and $\widehat{\theta}$.
 3.
Convert test statistics into pvalues via p_{ i }= 1  F_{0}(y$\widehat{\theta}$).
 4.
Estimate the PDF ${\widehat{f}}_{p}$(p) and CDF ${\widehat{F}}_{p}$(p) of the pvalues using the modified Grenander estimator (note that this requires ${\widehat{\eta}}_{0}$).
 5.Compute estimates of Fdr and fdr values based on pvalues:$\begin{array}{c}{\widehat{\text{fdr}}}_{p}(p)=\frac{{\widehat{\eta}}_{0}}{{\widehat{f}}_{p}(p)}\\ {\widehat{\text{Fdr}}}_{p}(p)=\frac{{\widehat{\eta}}_{0}p}{{\stackrel{\u02c6}{F}}_{p}(p)}\end{array}$
 6.Compute estimated Fdr and fdr values as a function of the original test statistics y:$\begin{array}{c}\widehat{\text{fdr}}(y)={\widehat{\text{fdr}}}_{p}(1{\widehat{F}}_{0}(y))\\ \widehat{\text{Fdr}}(y)={\widehat{\text{Fdr}}}_{p}(1{\widehat{F}}_{0}(y))\end{array}$
 7.Compute CDF and PDF on the yscale:$\begin{array}{c}\widehat{f}(y)={\widehat{\eta}}_{0}\frac{{\widehat{f}}_{0}(y)}{\widehat{\text{fdr}}(y)}\\ \widehat{F}(y)=1{\widehat{\eta}}_{0}\frac{1{\stackrel{\u02c6}{F}}_{0}(y)}{\widehat{\text{Fdr}}(y)}\end{array}$
 8.Estimate alternative subdensity:$\begin{array}{c}{\widehat{F}}_{A}(y)=\frac{\stackrel{\u02c6}{F}(y){\widehat{\eta}}_{0}{\stackrel{\u02c6}{F}}_{0}(y)}{1{\widehat{\eta}}_{0}}\\ {\widehat{f}}_{A}(y)=\frac{\widehat{f}(y){\widehat{\eta}}_{0}{\widehat{f}}_{0}(y)}{1{\widehat{\eta}}_{0}}\end{array}$
Results and discussion
Computer simulations for pvaluebased analyses
In order to assess the performance of the "fdrtool" algorithm it was compared with a number of other FDR procedures. Specifically, the R packages "fdrtool" version 1.2.4, "qvalue" version 1.1 [7], "locfdr" version 1.1–6 [4], "twilight" version 1.14.1 [30], "kerfdr" version 1.0.0 [23] and "nFDR" version 0.0 [24] were investigated.
First, FDR approaches based on pvalues were studied. As generative model pvalues were simulated from a mixture of the uniform U(0, 1) with either the truncated exponential density ${f}_{A}(p;a)=\frac{a}{\mathrm{exp}(a)1}\mathrm{exp}(a(1p))$ or with the uniform f_{ A }(p; a) = U (0, a). Sample size and mixture model parameter a were varied, and from each generated data set the proportion of null pvalues, and the squared error of the local FDR and the tail areabased FDR was estimated. The references for computing the squared error were the theoretical Fdr and fdr values derived from the assumed model.
The first column of Fig. 5 shows the accuracy in estimating η_{0}. Overall, the "kerfdr" and the "fdrtool" algorithms exhibit the smallest variability at the expense of a slightly biased estimate of η_{0}, especially for "model 1". In contrast, "qvalue" always produces nearly unbiased estimates but has a much higher variance. The "twilight" and "nFDR" are similar to "qvalue", but are less variable.
The second and third columns of Fig. 5 depict the error in the actually estimated Fdr and fdr values for various algorithms under the three model scenarios. In terms of correctly estimating fdr values all investigated packages with capability of computing local FDR (i.e. "fdrtool", "kerfdr", and "twilight") perform roughly equally well across all scenarios For "model 3" the "fdrtool" appears to have a slight advantage over the competing approaches. When comparing the accuracy of Fdr values "fdrtool" outperforms both "qvalue" and "nFDR", even though the differences are not large. The squared error of Fdr computed by "qvalue" and by "nFDR" exhibits more extreme outliers than those of "fdrtool".
Simulations and analysis of gene expression data for zscores
In a further simulation study estimation of FDR from zscores was considered with empirical null modeling. Specifically, data were simulated from a mixture of the normal distribution N(μ = 0, σ = 2) with the symmetric uniform alternatives U(10, 5) and U(5, 10).
The breast cancer data has size 3226. After mediancentering the data were again supplied to both the "fdrtool" and "locfdr" packages. Both algorithms indicated that there was overdispersion ($\widehat{\sigma}$ = 1.51 versus $\widehat{\sigma}$ = 1.55) and and the proportion of null values was estimated to be ${\widehat{\eta}}_{0}$ = 1. Correspondingly, for the BRCA data there were no significant zscores (note that this is in contrast to claims otherwise in [31]). In short, "fdrtool" and "locfdr" provide very similar analyzes both in terms of empirical null estimation and inferred fdr values.
Computational efficiency
Finally, the investigated FDR procedures were also compared in terms of computational efficiency. The (by a large margin) slowest program is "twilight". In contrast, the fastest algorithms are "fdrtool", "locfdr" and "qvalue", followed by "kerfdr" and "nFDR".
Conclusion
False discovery rate analysis is a key statistical innovation that has found widespread application in the study of highdimensional data. One of the intriguing aspects of FDR is that can be understood both from a frequentist and Bayesian perspective. This has lead to a plethora of FDR criteria and FDR inference procedures.
The goal for the development of the "fdrtool" procedure was to establish a common framework that brings together the most compelling features of existing FDR methods. Specifically, novel features of the proposed "fdrtool" algorithm include

a unified treatment of pvalues and other test statistics, with identical algorithms and learning procedures,

simultaneous and coherent estimation of both Fdr and fdr,

empirical null modeling for test statistics other than zscores,

a method for selecting the truncation point based on controlling FNDR, and

a simple semiparametric model using a modified Grenander density estimator.
Hence, "fdrtool" allows to compute local FDR values from pvalues but also Fdr values from zscores while taking account of an empirical null model. Despite the generality of the algorithm, it was shown that the accuracy of the algorithm is on par with the best competing yet more specialized FDR procedures. Moreover, the modular structure of the "fdrtool" procedure facilitates future extensions.
In summary, the "fdrtool" package and algorithm constitutes a comprehensive and featurerich tool for a wide range of FDRtype analyzes.
During revision a referee pointed out that the distribution of observed pvalues might be Ushaped [20]. This occurs, among other possibilities, if the null model is misspecified. As a result, the computed null pvalues do not follow a uniform distribution, and thus by definition are improper. "fdrtool" cannot be applied directly to improper pvalues, however, in these instances it might instead be preferable to conduct the FDR analysis on the level of the original test statistics.
Declarations
Acknowledgements
I thank Brit B. Turnbull (Stanford) for discussing the FDR algorithm implemented in "locfdr" and for kindly sending an early preprint. I thank Florian Leitenstorfer (Munich) for discussion and insights concerning monotone regression. I also would like to thank the three anonymous referees for their very helpful and detailed comments.
Authors’ Affiliations
References
 Schweder T, Spjøtvoll E: Plots of p values to evaluate many tests simultaneously. Biometrika 1982, 69: 493–502.View ArticleGoogle Scholar
 Benjamini Y, Hochberg Y: Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Statist Soc B 1995, 57: 289–300.Google Scholar
 Schäfer J, Strimmer K: An empirical Bayes approach to inferring largescale gene association networks. Bioinformatics 2005, 21: 754–764. 10.1093/bioinformatics/bti062View ArticlePubMedGoogle Scholar
 Efron B: Largescale simultaneous hypothesis testing: the choice of a null hypothesis. J Amer Statist Assoc 2004, 99: 96–104. 10.1198/016214504000000089View ArticleGoogle Scholar
 Langaas M, Lindqvist BH, Ferkingstad E: Estimating the proportion of true null hypotheses, with application to DNA microarray data. J R Statist Soc B 2005, 67: 565–572. 10.1111/j.14679868.2005.00515.xView ArticleGoogle Scholar
 Broberg P: A comparative review of estimates of the proportion unchanged genes and the false discovery rate. BMC Bioinformatics 2005, 6: 199. 10.1186/147121056199PubMed CentralView ArticlePubMedGoogle Scholar
 Storey JD, Tibshirani R: Statistical significance for genomewide studies. Proc Natl Acad Sci USA 2003, 100: 9440–9445. 10.1073/pnas.1530509100PubMed CentralView ArticlePubMedGoogle Scholar
 Efron B: Microarrays, empirical Bayes, and the twogroups model. Statistical Science 2008., 23: to appear.Google Scholar
 Genovese C, Wassermann L: Operating characteristics and extensions of the false discovery rate procedure. J R Statist Soc B 2002, 64: 499–517. 10.1111/14679868.00347View ArticleGoogle Scholar
 Bonferroni CE: Il calcolo delle assicurazioni su gruppi di teste. Studi in Onore del Professore Salvatore Ortu Carboni, Rome 1935, 13–60.Google Scholar
 Storey JD: The positive false discovery rate: A Bayesian interpretation and the qvalue. Ann Statist 2003, 31: 2013–2035. 10.1214/aos/1074290335View ArticleGoogle Scholar
 Storey JD: A direct approach to false discovery rates. J R Statist Soc B 2002, 64: 479–498. 10.1111/14679868.00346View ArticleGoogle Scholar
 Efron B, Tibshirani R, Storey JD, Tusher V: Empirical Bayes analysis of a microarray experiment. J Amer Statist Assoc 2001, 96: 1151–1160. 10.1198/016214501753382129View ArticleGoogle Scholar
 Efron B: Robbins, empirical Bayes, and microarrays. Annals of Statistics 2003, 31: 366–378. 10.1214/aos/1051027871View ArticleGoogle Scholar
 Aubert J, BarHen A, Daudin JJ, Robin S: Determination of the differentially expressed genes in microarray experiments using local FDR. BMC Bioinformatics 2004, 5: 125. 10.1186/147121055125PubMed CentralView ArticlePubMedGoogle Scholar
 Efron B: Correlation and largescale simultaneous significance tesing. J Amer Statist Assoc 2007, 102: 93–103. 10.1198/016214506000001211View ArticleGoogle Scholar
 Sun W, Cai TT: Oracle and adaptive compound decision rules for false discovery control. J Amer Statist Assoc 2007, 102: 901–912. 10.1198/016214507000000545View ArticleGoogle Scholar
 R Development Core Team: R: A language and environment for statistical computing.R Foundation for Statistical Computing, Vienna, Austria; 2007. [http://www.Rproject.org]Google Scholar
 Strimmer K: fdrtool: a versatile R package for estimating local and tail areabased false discovery rates. Bionformatics 2008, 24: 1461–1462. 10.1093/bioinformatics/btn209View ArticleGoogle Scholar
 Pounds S, Cheng C: Robust estimation of the false discovery rate. Bioinformatics 2006, 22: 1979–1987. 10.1093/bioinformatics/btl328View ArticlePubMedGoogle Scholar
 Efron B, Tibshirani R: Using specially designed exponential families for density estimation. Ann Statist 1998, 24: 2431–2461.Google Scholar
 McLachlan GJ, Bean RW, Jones LBT: A simple implementation of a normal mixture approach to differential gene expression in multiclass microarrays. Bioinformatics 2006, 22: 1608–1615. 10.1093/bioinformatics/btl148View ArticlePubMedGoogle Scholar
 Robin S, BarHen A, Daudin JJ, Pierre L: A semiparametric approach for mixture models: application to local false discovery rate estimation. Comput Statist Data Analysis 2007, 51: 5483–5493. 10.1016/j.csda.2007.02.028View ArticleGoogle Scholar
 Guan Z, Wu B, Zhao H: Nonparametric estimator of false discovery rate based on Bernstein polynomials. Statistica Sinica 2008, in press.Google Scholar
 Grenander U: On the theory of mortality measurement, Part II. Skan Aktuarietidskr 1956, 39: 125–153.Google Scholar
 Robertson T, Wright FT, Dykstra RL: Order restricted statistical inference. John Wiley and Sons; 1988.Google Scholar
 Pounds S, Morris SW: Estimating the occurrence of false positives and false negatives in microarray studies by approximating and partitioning the empirical distribution of pvalues. Bioinformatics 2003, 19: 1236–1242. 10.1093/bioinformatics/btg148View ArticlePubMedGoogle Scholar
 Turnbull BB: Optimal estimation of false discovery rates. Tech rep Stanford University; 2007. [http://www.stanford.edu/~bkatzen/optimalFDR.pdf]Google Scholar
 Efron B: Size, power and false discovery rates. Ann Statist 2007, 35: 1351–1377. 10.1214/009053606000001460View ArticleGoogle Scholar
 Scheid S, Spang R: A stochastic downhill search algorithm for estimating the local false disovery rate. IEEE T Comp Biol Bioinf 2004, 1: 98–108. 10.1109/TCBB.2004.24Google Scholar
 Jin J, Cai TT: Estimating the null and the proportion of nonnull effects in largescale multiple comparisons. J Amer Statist Assoc 2007, 102: 495–506. 10.1198/016214507000000167View ArticleGoogle Scholar
 Dalmasso C, Bröet P, Moreau T: A simple procedure for estimating the false discovery rate. Bioinformatics 2005, 21: 660–668. 10.1093/bioinformatics/bti063View ArticlePubMedGoogle Scholar
 Liao JG, Lin Y, Selvanayagam ZR, Shih WJ: A mixture model for estimating the local false discovery rate in DNA microarray analysis. Bioinformatics 2004, 20: 2694–2701. 10.1093/bioinformatics/bth310View ArticlePubMedGoogle Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.