Generalized shrinkage F-like statistics for testing an interaction term in gene expression analysis in the presence of heteroscedasticity
© Yang et al; licensee BioMed Central Ltd. 2011
Received: 10 June 2010
Accepted: 1 November 2011
Published: 1 November 2011
Many analyses of gene expression data involve hypothesis tests of an interaction term between two fixed effects, typically tested using a residual variance. In expression studies, the issue of variance heteroscedasticity has received much attention, and previous work has focused on either between-gene or within-gene heteroscedasticity. However, in a single experiment, heteroscedasticity may exist both within and between genes. Here we develop flexible shrinkage error estimators considering both between-gene and within-gene heteroscedasticity and use them to construct F-like test statistics for testing interactions, with cutoff values obtained by permutation. These permutation tests are complicated, and several permutation tests are investigated here.
Our proposed test statistics are compared with other existing shrinkage-type test statistics through extensive simulation studies and a real data example. The results show that the choice of permutation procedures has dramatically more influence on detection power than the choice of F or F-like test statistics. When both types of gene heteroscedasticity exist, our proposed test statistics can control preselected type-I errors and are more powerful. Raw data permutation is not valid in this setting. Whether unrestricted or restricted residual permutation should be used depends on the specific type of test statistic.
The F-like test statistic that uses the proposed flexible shrinkage error estimator considering both types of gene heteroscedasticity and unrestricted residual permutation can provide a statistically valid and powerful test. Therefore, we recommended that it should always applied in the analysis of real gene expression data analysis to test an interaction term.
The regulation of gene expression starts when a cell's DNA is transcribed into mRNA. The simultaneous expression profiles of many genes under different circumstances can provide insight into physiological processes. Using modern technologies in gene expression experiments such as oligonucleotide arrays , and cDNA spotted arrays , many scientists have made novel discoveries about complex biological processes of yeast [3, 4], drosophila , mice , humans , and other species. Recently one such study also included RNA-seq . Statistical methodologies and issues involved in microarray data analysis have been widely reviewed [9–12], and it is expected that many of the same issues will need to be addressed with RNA-seq.
The analysis of variance (ANOVA) model is a popular statistical modeling method for the analysis of microarrays. Since its introduction by Kerr et al. , it has been extensively examined for use in this setting [14–21]. Kerr et al. constructed an ANOVA model that included the gene effect as a fixed effect. This model assumes identically and independently distributed residual errors across genes. The advantage of this model is that the large number of genes involved in a microarray experiment results in huge degrees of freedom for the error estimate, which can lead to a very powerful test. However, the common assumption of homoscedasticity may not hold true in this setting . One alternative is to use an ANOVA model for each gene, but the resulting test statistics from gene-specific models may have limited power because the biological sample size for each gene in a microarray experiment is usually small.
To address this problem of limited power, researchers have proposed other methods for obtaining more information across genes, ranging from a simple equal-weighted average of a gene-specific error estimate and the global average of all gene-specific error estimates (F2 statistic proposed by Wu et al.  to empirical Bayesian modeling of all gene-specific errors [23–26]. Other variations [27–29] used different variance modeling strategies to address the heteroscedasticity problem, but no clear winner has emerged . Huang and Liu  extended the test statistics proposed by Cui et al.  by assuming a normal distribution on the mean and then deriving an empirical Bayes likelihood ratio test. The resulting test statistic shrinks both the mean and variances.
In addition to the problem of between-gene heteroscedasticity, we must also be concerned with within-gene heteroscedasticity. For example, in the study of simple differential gene expression between a treatment group and a control group, the variance in the treatment arm may differ from that in the control arm. Some approaches to this problem include a general Bayesian framework to model heteroscedastic error in a single generalized linear mixed model setting  and a structural model placed on the error variances specific to each gene and treatment combination .
As gene expression studies become more popular, the complexity of the experiment increases. Instead of only simple treatment and control experiments, two or more factor experiments are being conducted. This increase in experiment complexity has led to many scientific questions involving the hypothesis testing of an interaction between two factors. For example, testing a probe by genotype interaction can result in inferences about polymorphism in the probe, such as single nucleotide polymorphism (SNP) and insertion-deletion (indel) [34–37]; testing a probe by sex can imply that alternative splicing occurs between male and female subjects ; and in pharmacogenomic studies, testing the genotype-drug/treatment or genotype-disease interaction may be of interest . Thus far, all the development of ANOVA methods for microarray studies has focused on tests of main effects.
Here, a generalized shrinkage estimator incorporating both within- and between-gene heteroscedasticities is developed (see Lehmann and Cesella  for a review of shrinkage estimation). In any given experiment, both within-gene and between-gene heteroscedasticity may exist; thus, taking these possibilities into account should lead to an improved test statistic. Moreover, given the increasing complexity of recent studies and the burgeoning interest in hypotheses that involve interactions, we focus on an improved shrinkage-based F-test for interaction terms.
Here we develop new shrinkage estimates for the error term and show how to use these estimates to construct F-like statistics. We then estimate the null distribution of these statistics by using permutation tests.
Shrinkage error estimators
i = 1,...,I, k = 1,..., K, and let be the true variance of gene i in group k. When the experimental design is balanced, is the residual mean square for gene i in group k and , where ν represents the degrees of freedom for the error estimates.
Specific values for each gene-group combination
Gene-specific values that are the same across all other groups
Group-specific values that are the same across genes but different across groups
A single point representing the underlying common error
Correspondingly, these targets are correct when (1) there are both within-gene and between-gene heteroscedasticity; (2) there is only between-gene heteroscedasticity; (3) there is only within-gene heteroscedasticity; and (4) all error variances are identical. We now develop a generalized shrinkage error estimator using these four shrinkage targets.
where represents the gene-specific mean differences, and models different means with respect to different classes of the subgroups.
the empirical Bayes estimator for τ2 is .
where X i is the residual variance estimate from a gene-specific model, and m and σ2 are the mean and variance of log . The underlying assumption for this estimator is that there is no between-gene heteroscedasticity, as this estimator shrinks every gene-specific error estimator toward one target. Therefore, it will overshrink the gene-specific error estimates when gene heteroscedasticity exists. In comparison, generalized shrinkage error estimators are flexible in terms of incorporating a different type of heteroscedasticity. Some degrees of freedom are used for incorporating the heteroscedasticity. However, the gain is that the error estimator is then closer to the underlying distribution and should lead to better performance of the resultant F-like test statistics as shown in the results section.
Results from raw data permutation
Shrinkage F-like statistics
The error variance estimator in this test uses data from only gene i. In oligonucleotide mi-croarray models, the degrees of freedom for the error estimate can be small because the sample size of RNA is usually small, and hence the power of F1 can be limited.
For the proposed generalized shrinkage F-like test statistics, the null distributions are not known named distributions. Therefore, an empirical approach such as a permutation test can be used to estimate the null distributions. The permutation test for interaction is complicated, because there is no exact permutation test for such a purpose . We therefore must consider an approximate permutation method for testing an interaction term in a crossed fixed/mixed model [44, 45].
Permutation approaches developed previously focused on a single ANOVA model. In the typical gene expression study, thousands of ANOVA models are considered simultaneously. The additional complexity of the shrinkage F-like statistics indicates that Monte Carlo studies are needed to investigate the performance of residual permutation and raw data permutation, with restrictions or not, in a gene-expression analysis. The choice of permutation procedures is critical for assessing the performance of a test statistic.
For all the modified F-like statistics presented in the previous section, the null distributions can only be approximated empirically, but permutation procedures can be used to find the approximate null distribution of all the F and F-like statistics. The important issues in performing a permutation analysis include the choice of the exchangeable units under the null hypothesis, the choice of using restricted permutation or not, and the choice of residual permutation or raw data permutation. These choices influence the power of a test statistic.
Residual permutation using residuals from a reduced model and unrestricted raw data permutation can be used to approximate the null distribution of a statistic for testing an interaction term . When using F1 to test an interaction term in a single ANOVA model, the residual permutation leads to a more powerful test than unrestricted raw data permutation . However, in gene expression analysis, thousands of gene-specific ANOVA models are simultaneously considered, and for a particular gene-specific ANOVA model, information from other gene-specific ANOVA models is used to construct the shrinkage error estimate. Hence, both residual permutation and raw data permutation were investigated. Furthermore, both restricted and unrestricted permutations were studied, because the permutation units are exchangeable only within each particular group when within-gene heteroscedasticity is present across those subgroups.
The properties of this shrinkage estimator are compared with those of other existing F and F-like statistics that have been proposed and described in the "Shrinkage F-like statistics" section.
The purpose of these simulation studies was to compare the performances of F1, F2, F3, F Cui , F Gen , F Gen-gene , and F Gen-grp in terms of type I error and power and to compare the results of a particular F-like statistic using four different permutation strategies: restricted/unrestricted residual permutation and restricted/unrestricted raw data permutation.
In these simulation studies, 100 genes with two probes for each gene and three replicates from each of two lines were simulated to mimic a split-plot design in a general oligonu-cleotide microarray experiment. The gene-specific ANOVA model in which data were generated from the model, y plr = P p + L l + RL rl + PL pl + ϵ plr , wp = 1, 2, l = 1,2, r = 1,2,3, where P, L, RL, and PL represent probe, line, replicates from a particular line, and the interaction between probe and line, respectively.
Replicates were nested within each line, and RL is usually treated as a random effect during the model-fitting procedure, which results in a correlation between probes from the same biological sample. In the simulated data sets, the correlation between genes was 0. As many as 900 simulation runs were carried out to compare the performances of F1, F2, F3, F Cui , F Gen , F Gen-gene , and F Gen-grp based on different permutation procedures. The four permutations tested were unrestricted residual permutation, restricted residual permutation with respect to each line, unrestricted raw data permutation, and restricted raw data permutation with respect to each line. The residuals permuted were from a reduced fixed model with fixed effects for only line and probe.
Results from residual permutation
Further simulations to compare the rejection rates were conducted. Only results from residual permutation are shown because it was found that raw data permutation was less powerful than residual permutation. This is consistent with the findings of Anderson and Ter Braak . Figure 1 shows the estimated average null hypothesis rejection rate curves from all F-like statistics and both restricted and unrestricted residual permutation procedures. The x-axis represents the average null hypothesis rejection rate using F1 and the tabulated p-values. The solid line shows that the corresponding statistic controls the prespecified CWER, and the dashed line shows that the corresponding CWER was inflated. In general, restricted residual permutation is less powerful than unrestricted residual permutation. For example, the power of all statistics from unrestricted residual permutation almost doubled in some cases where heteroscedasticity existed.
The data used in this study are from a gene expression comparison study between D. melanogaster and D. simulans . Expression of 10 genotypes of each species was measured in male flies. In D. simulans, each genotype was measured separately, and in D. melanogaster, a pool of 10 genotypes was measured. All genotypes (individual or pooled) were independently isolated and hybridized three times. The goal of the original study was to provide a genome-wide approach to identifying candidate genes potentially responsible for adaptation and speciation in D. simulans and D. melanogaster. In this study, we focus on identifying sequence differences between genotypes in D. simulans based on hybridization profiles. Within-gene heteroscedasticity is expected because the genotypes come from different lines. The proposed generalized shrinkage F-like test statistics F Gen , F Gen-gene , and F Gen-grp were compared with F2, F3 with restricted residual permutation, which could control prespecified CWER for any variance structure in simulation studies. Furthermore, Smyth's moderated F-test statistic  without multiple testing adjustment and controlling the false discovery rate (FDR) at 5% were used for comparison. As the main interest is in sequence difference, the focus is on the test of interaction between line and probe. The split plot model described above is used. SAS program codes are included in the additional files (additional file 1 and additional file 2).
Probe sets with significant line*probe terms found by F-like test statistics and appropriate residual permutation procedures and Smyth's moderated F-test statistic
Number of probe sets found
True false discovery rate
F Gen grp
moderatedF - 1
moderatedF - 2
For gene expression analysis, ANOVA models have been a popular modeling technique. Based on ANOVA models, flexible shrinkage F-like test statistics were developed to account for both the within-gene and between-gene heteroscedasticities. The emphasis here is on testing an interaction term, as this case is of increasing interest to biologists, and there is no clear existing theory on the most powerful, valid approach for such statistics. For all F-like statistics studied here, their null distributions were approximated empirically through permutations. Four different permutation procedures were investigated for eight different F-like statistical tests of the interaction term.
As expected, we found that when an error estimator overshrinks, the resulting F-like statistic cannot control the prespecified CWER. For example, F Gen-gene is an over-shrinkage error estimator when there is within-gene heteroscedasticity. As a result, compared with generalized shrinkage F-like statistics, it is not valid when within-gene heteroscedasticity exists. Undershrinkage is also important, as it will lead to a conservative test and lower power. This is clearly demonstrated when the common error can be assumed and the most powerful valid test is F Gen-grp .
The most striking result was the impact of the permutation procedures. Although this was not completely unexpected [43–45], the effect of the permutation procedures is dramatic and worthy of special attention. Unrestricted raw data permutation could not control prespeci-fied CWER when there was within-gene heteroscedasticity. Restricted raw data permutation could be used, but it was less powerful than residual permutation. Also consistent with findings from Anderson and Ter Braak , restricted permutations are less powerful than unrestricted permutations. However, unrestricted permutations are valid only for a common error and when between-gene heteroscedasticity exists for our proposed shrinkage statistics; they are not valid in combination with F2, F3, or F Cui . For F Gen-grp , the unrestricted permutation can also be used in cases having within-gene heteroscedasticity, while only F Gen is valid with unrestricted permutation in all cases in terms of controlling prespecified CWER. Interestingly, the power gain from using the correct shrinkage target F Gen-grp rather than F Gen is far less than that of using unrestricted permutation. The result is that F3 is never the most powerful choice when testing an interaction term.
The correct shrinkage target can lead to the most powerful test statistic. As one of the reviewers suggested, a statistical test may be applied to help pick the best shrinkage target before obtaining shrinkage error estimates. However, this extra testing step may inflate the CWER of the test statistic when there is gene heteroscedasticity. For example, when there are both types of gene heteroscedasticities, it is possible that the above test suggests only within-gene heteroscedasticities exist, and F Gen-grp is shown to inflate the CWER. There is minimal penalty to using the shrinkage estimator we propose, so we recommend setting the shrinkage target in the full space spanned by group and gene and using unrestricted permutation to compensate for the possible power loss in fewer degrees of freedom left for estimating the errors.
The proposed generalized shrinkage F-like statistic with shrinkage targets located in a space spanned by gene and another group, F Gen , with unrestricted residual permutation is always valid in terms of having a prespecified CWER. This statistic has reasonable power in most cases; thus, it is generally recommended to be applied to test an interaction term in the analysis of real gene expression data.
List of abbreviations
comparison-wise error rate
false discovery rate
insertion and deletion
single nucleotide polymorphism
We thank Brandon Walts for identifying true SNP positions; Angela J. McArthur and David R. Galloway for their help in scientific editing; associate editor and three reviewers for their constructive comments that much improved this manuscript. This research was supported by NIH 1R01GM077618 (McIntyre), NIH 1R01GM081704 (Casella).
- Fodor SPA: Massively parallel genomics. Science 1997, 277: 393–395. 10.1126/science.277.5324.393View ArticleGoogle Scholar
- Schena M, Shalon D, Heller R, Chai A, Brown PO, Davis RW: Parallel human genome analysis: microarray-based expression monitoring of 1000 genes. Proceedings of National Academy Science 1996, 93: 10614–10619. 10.1073/pnas.93.20.10614View ArticleGoogle Scholar
- Galitski T, Saldanha AJ, Styles CA, Lander ES, Fink GR: Ploidy regulation of gene expression in yeast. Science 1999, 285: 251–254. 10.1126/science.285.5425.251View ArticlePubMedGoogle Scholar
- Tu BP, Kudlicki A, Rowicka M, McKnight SL: Logic of the yeast metabolic cycle: Temporal compartmentalization of cellular processes. Science 2005, 310: 1152–1158. 10.1126/science.1120499View ArticlePubMedGoogle Scholar
- White KP, Rifkin SA, Hurban P, Hogness DS: Microarray analysis of Drosophila development during metamorphosis. Science 1999, 286: 2179–2184. 10.1126/science.286.5447.2179View ArticlePubMedGoogle Scholar
- Chabas D, Baranzini SE, Mitchell D, Bernard CCA, Rittling SR, Denhardt DT, Sobel RA, Lock C, Karpuj M, Pedotti R, Heller R, Oksenberg JR, Steinman L: The influence of the proinflammatory cytokine, Osteopontin, on autoimmune demyelinating disease. Science 2001, 294: 1731–1735. 10.1126/science.1062960View ArticlePubMedGoogle Scholar
- Sebat J, Lakshmi B, Troge J, Alexander J, Young J, Lundin P, Maner S, Massa H, Walker M, Chi MY, Navin N, Lucito R, Healy J, Hicks J, Ye K, Reiner A, Gilliam TC, Trask B, Patterson N, Zetterberg A, Wigler M: Large-scale copy number polymorphism in the human genome. Science 2005, 305: 525–528.View ArticleGoogle Scholar
- Blekhman R, Marioni JC, Zumbo P, Stephens M, Gilad Y: Sex-specific and lineage-specific alternative splicing in primates. Genome Research 2010, 20(2):180–189. 10.1101/gr.099226.109PubMed CentralView ArticlePubMedGoogle Scholar
- Butte A: The use and analysis of microarray data. Nature Reviews 2002, 1: 951–960. 10.1038/nrd961PubMedGoogle Scholar
- Churchill GA: Fundamentals of experimental design for cDNA microarrays. Nature Genetics 2002, 32: 490–495. 10.1038/ng1031View ArticlePubMedGoogle Scholar
- Craig BA, Black MA, Doerge RW: Gene expression data: The technology and statistical analysis. Journal of Agricultural, Biological, and Environmental Statistics 2003, 8(1):1–28. 10.1198/1085711031256View ArticleGoogle Scholar
- Allison DA, Cui X, Page GP, Sabripour M: Microarray data analysis: from disarray to consolidation and consensus. Nature Reviews Genetics 2006, 7: 55–65. 10.1038/nrg1749View ArticlePubMedGoogle Scholar
- Kerr MK, Martin M, Churchill GA: Analysis of variance for gene expression microarray data. Journal of Computational Biology 2000, 7: 819–837. 10.1089/10665270050514954View ArticlePubMedGoogle Scholar
- Kerr MK, Churchill GA: Statistical design and the analysis of gene expression microarrays. Genetical Research 2001, 77: 123–128.PubMedGoogle Scholar
- Kerr MK, Churchill GA: Experimental design for gene expression microarrays. Biostatistics 2001, 2: 183–201. 10.1093/biostatistics/2.2.183View ArticlePubMedGoogle Scholar
- Pritchard CC, Hsu L, Delrow J, Nelson PS: Project normal: Defining normal variation in mouse gene expression. Proceedings of the National Academy of SciencesUSA 2001, 98: 13266–13271. 10.1073/pnas.221465998View ArticleGoogle Scholar
- Wolfinger RD, Gibson G, Wolfinger ED, Bennett L, Hamadeh H, Bushel P, Ashfari C, Paules RS: Assessing gene significance from cDNA microarray expression data via mixed models. Journal of Computational Biology 2001, 8(6):625–637. 10.1089/106652701753307520View ArticlePubMedGoogle Scholar
- Kerr MK, Afshari CA, Bennett L, Bushel P, Martinez J, Walker NJ, Churchill GA: Statistical analysis of a gene expression microarray experiment with replication. Statistica Sinica 2002, 12: 203–217.Google Scholar
- Wu H, Kerr MK, Cui XQ, Churchill GA: MAANOVA: A Software package for the analysis of spotted cDNA microarray experiments, In. In The analysis of gene expression data: methods and software. Springer; 2002:313–341.Google Scholar
- Chu T, Weir B, Wolfinger R: A systematic statistical linear modeling approach to oligonucleotide array experiments. Mathematical Biosciences 2002, 176: 35–51. 10.1016/S0025-5564(01)00107-9View ArticlePubMedGoogle Scholar
- Wayne ML, Pan YJ, Nuzhdin SV, McIntyre LM: Additivity and transacting effects on gene expression in male Drosophila simulans . Genetics 2004, 168: 1413–1420. 10.1534/genetics.104.030973PubMed CentralView ArticlePubMedGoogle Scholar
- Cui X, Churchill GA: Statistical tests for differential expression in cDNA microarray experiments. Genome Biology 2003, 4(4):201.View ArticleGoogle Scholar
- Baldi P, Long AD: A Bayesian framework for the analysis of microarray expression data: Regularized t-test and statistical inferences of gene changes. Bioinformatics 2001, 17: 509–519. 10.1093/bioinformatics/17.6.509View ArticlePubMedGoogle Scholar
- Lönnstedt I, Speed T: Replicated microarray data. Statistica Sinca 2002, 12: 31–46.Google Scholar
- Smyth GK: Linear models and empirical Bayes methods for assessing differential expression in microarray experiments. Statistical Applications in Genetics and Molecular Biology 2004., 3: No. 1, Article 3 No. 1, Article 3Google Scholar
- Tong TJ, Wang YD: Optimal shrinkage estimation of variances with applications to microarray data analysis. Journal of the American Statistical Association 2007, 102: 113–122. 10.1198/016214506000001266View ArticleGoogle Scholar
- Tusher VG, Tibshirani R, Chu G: Significance analysis of microarrays applied to ionizing radiation response. The Preceedings of National Academy Science 2001, 989: 5116–5121.View ArticleGoogle Scholar
- Cui X, Hwang JTG, Qiu J, Blades NJ, Churchill GA: Improved statistical tests for differential gene expression by shrinking variance components estimates. Biostatis-tics 2005, 6: 59–75. 10.1093/biostatistics/kxh018View ArticleGoogle Scholar
- Feng S, Wolfinger RD, Chu TM, Gibson GC, McGraw LA: Empirical Bayes analysis of variance component models for microarray data. Journal of Agricultural, Biological & Environmental Statistics 2006, 1113: 197–190.View ArticleGoogle Scholar
- Kim SY, Lee JW, Sohn IS: Comparison of various statistical methods for identifying differential gene expression in replicated microarray data. Statistical Methods in Medical Research 2006, 15: 3–20. 10.1191/0962280206sm423oaView ArticlePubMedGoogle Scholar
- Hwang JTG, Liu P: Optimal tests shrinking both means and variances applicable to microarray data analysis. In preprint 2007–03. Department of Statistics, Iowa State University, IA; 2007.Google Scholar
- Kizilkaya K, Tempelman RJ: A general approach to mixed effects modeling of residual variances in generalized linear mixed models. Genetics Selection Evolution 2005, 37: 31–56. 10.1186/1297-9686-37-1-31View ArticleGoogle Scholar
- Jaffrezic F, Marot G, Degrelle S, Isabelle H, Foulley JL: A structural mixed model for variances in differential gene expression studies. Genetical Research 2007, 89(1):19–25. 10.1017/S0016672307008646View ArticlePubMedGoogle Scholar
- Rostoks N, Borevitz JO, Hedley PE, Russell J, Mudie S, Morris J, Cardle L, Marshall DF, Waugh R: Single-feature polymorphism discovery in the Barley transcriptome. Genome Biology 2005, 6: R54. 10.1186/gb-2005-6-6-r54PubMed CentralView ArticlePubMedGoogle Scholar
- Kirst M, Caldo R, Casati P, Tanimoto G, Walbot V, Wise RP, Buckler ES: Genetic iversity contribution to errors in short oligonucleotide microarray analysis. Plant Biotechnology Journal 2006, 4: 489–498.PubMedGoogle Scholar
- Zhang X, Shiu SH, Cal A, Borevitz JO: Global analysis of genetic, epigenetic and transcriptional polymorphisms in Arabidopsis Thaliana Using whole genome tiling arrays. PLoS Genetics 2008, 4(3):e1000032. 10.1371/journal.pgen.1000032PubMed CentralView ArticlePubMedGoogle Scholar
- Zhang X, Borevitz JO: Global Analysis of Allele-specific Expression in Arabidopsis Thaliana. Genetics 2009, 182(4):943–954. 10.1534/genetics.109.103499PubMed CentralView ArticlePubMedGoogle Scholar
- McIntyre LM, Bono LM, Genissel A, Westerman R, Junk D, Telonis-Scott M, Harshman L, Wayne ML, Kopp A, Nuzhdin SV: Sex specific expression of alternative transcripts in Drosophila. Genome Biology 2006, 7: R79. 10.1186/gb-2006-7-8-r79PubMed CentralView ArticlePubMedGoogle Scholar
- Kelly P, Zhou YH, Whitehead J, Stallard N, Bowman C: Sequentially testing for a gene-drug interaction in a genomewide analysis. Statistics in Medicine 2008, 27: 2022–2034. 10.1002/sim.3059View ArticlePubMedGoogle Scholar
- Lehmann EL, Casella G: Theory of Point Estimation. 2nd edition. New York: Springer-Verlag; 1998.Google Scholar
- Pounds S: Computational enhancement of a shrinkage-based analysis of variance F-test proposed for differential gene expression analysis. Biostatistics 2007, 83: 505–506.View ArticleGoogle Scholar
- Neter J, Wasserman W, Kutner MH: Applied Linear Statistical Models: Regression, Analysis of Variance, and Experimental Designs. 3rd edition. Irwin, Inc; 1990.Google Scholar
- Edgington ES: Randomization Tests. 3rd edition. Marcel Dekker, New York; 1995. (1995) (1995)Google Scholar
- Anderson MJ, Ter Braak CJF: Permutation tests for multi-factorial analysis of anova. Journal of Statistical Computation and Simulation 2003, 732: 85–113.View ArticleGoogle Scholar
- Churchill GA, Doerge RW: Naive application of permutation testing leads to nflated type I error rates. Genetics 2008, 178: 609–610. 10.1534/genetics.107.074609PubMed CentralView ArticlePubMedGoogle Scholar
- Nuzhdin SV, Wayne ML, Harmon KL, McIntyre LM: Common pattern of evolution of gene expression level and protein sequence in drosophila. Molecular Biology and Evolution 2004, 21: 1308–1317. 10.1093/molbev/msh128View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.