Algorithm-driven Artifacts in median polish summarization of Microarray data
© Giorgi et al. 2010
Received: 5 January 2010
Accepted: 11 November 2010
Published: 11 November 2010
Skip to main content
© Giorgi et al. 2010
Received: 5 January 2010
Accepted: 11 November 2010
Published: 11 November 2010
High-throughput measurement of transcript intensities using Affymetrix type oligonucleotide microarrays has produced a massive quantity of data during the last decade. Different preprocessing techniques exist to convert the raw signal intensities measured by these chips into gene expression estimates. Although these techniques have been widely benchmarked in the context of differential gene expression analysis, there are only few examples where their performance has been assessed in respect to coexpression-based studies such as sample classification.
In the present paper we benchmark the three most used normalization procedures (MAS5, RMA and GCRMA) in the context of inter-array correlation analysis, confirming and extending the finding that RMA and GCRMA consistently overestimate sample similarity upon normalization. We determine that median polish summarization is responsible for generating a large proportion of these over-similarity artifacts. Furthermore, we show that most affected probesets show also internal signal disagreement, and tend to be composed by individual probes hitting different gene transcripts. We finally provide a correction to the RMA/GCRMA summarization procedure that massively reduces inter-array correlation artifacts, without affecting the detection of differentially expressed genes.
We propose tRMA as a modification of RMA to normalize microarray experiments for correlation-based analysis.
High density oligonucleotide microarrays are widely used in many areas of biological research for quantitative, high-throughput measurements of gene expression. Although ultra-deep sequencing techniques promise to replace them in the near future , it would be a mistake to ignore the biological importance of the massive quantity of data already produced through this platform. Publicly available databases alone store a huge (and growing) quantity of microarray experiments (e.g. 338947 samples in Gene Expression Omnibus  and 251711 in ArrayExpress ), comprising hundreds of different species.
Among microarrays, the single-channel Affymetrix GeneChip platform  is by far the most popular (for instance, in Gene Expression Omnibus they represent 97.9% of all arrays available for Arabidopsis thaliana, and 99.0% for Homo sapiens). In this technology each transcript is typically measured by a set of 11-20 pairs of 25 bases-long probes, collectively referred to as "probeset".
For every "perfect match" probe (PM), Affymetrix chips contain a "mismatch" counterpart (MM), with a single nucleotide change in the middle of the PM probe sequence. The role of MM probes, located adjacent to the respective PM, is to measure probe-specific background signal associated to any perfect-match signal intensity.
In general, the process of obtaining a single gene expression value out of raw probe intensity measurements is called "microarray preprocessing". Three steps are usually required: background correction, normalization and summarization. Many different methods or combinations of methods were proposed over the years [5, 6].
The most popular manufacturer-provided method, MAS5 , uses a scale normalization approach, then corrects the background by subtracting the mean intensity of the lowest 2% spots in every microarray region, and then MM intensities from the respective PM ones. Wherever MM intensity is higher than a PM one, in order to avoid negative signal intensities typical of the MAS5 predecessor, MAS4 [4, 8], MAS5 replaces the MM signal with an "idealized mismatch" value (IM) derived from other values in the same probeset. To extract final probeset intensities, MAS5 calculates a robust average (Tukey's biweight) of all the contained probes.
Many alternative techniques have challenged MAS5 supremacy for preprocessing. Being a single-array technique, MAS5 doesn't model probes' behaviour across different samples, and therefore suffers from high variance and is theoretically less robust than multi-array algorithms [9, 10].
Two of the most popular multi-array normalization techniques are RMA  and GCRMA . RMA doesn't use information contained in MM probes, and calculates background signal by performing a modelled global correction of all PM intensities. Then it applies a quantile normalization step and a median polish summarization, which accounts for probe intensities over multiple arrays. GCRMA applies the same normalization and summarization steps as RMA, but it differs in the background correction method, which is based on the probe sequence. Other multi-array methods which don't discard MM intensities exist, one of them being dChip . However, in the present paper we will focus on RMA, GCRMA and MAS5, which are possibly the most popular microarray normalization methods [13, 14]. Their popularity is illustrated by the fact that they are the most popular normalization techniques in online databases .
Most benchmarks have tried to assess the quality of different preprocessing methods in differential gene expression scenarios, the original purpose for which microarrays were developed . To do so, golden set spike-in samples were used, with known concentrations of transcripts [17, 6], or Real Time PCR measurements were performed for comparison . The outcome of these benchmarks has not identified any technique as the top performer, although single-array techniques such as MAS5 have been outperformed by multi-array ones such as RMA [9, 18, 19].
However, many different approaches to biological investigation have relied on microarrays, ranging from gene and sample clustering  to gene-gene network reverse-engineering , from sample classification  to global transcript models . The field of microarray data correlation and clustering based on the principle of coexpression has developed at a quite considerable pace ; despite this, the effects of preprocessing on coexpression analyses have been generally overlooked, with a few exceptions.  used bacterial operons to validate the different normalization techniques for correlation analysis and concluded that a combination of different methods works best. On the other hand,  have pointed out how the use of the multi-array techniques RMA and GCRMA can yield inter-array correlation artifacts and generally lower quality networks than the older MAS5. In particular, a specific step in GCRMA background correction (the gene-specific binding correction, or GSB) has been identified as partially responsible for the spurious correlations generated by GCRMA. Notably however, the correction of this step is not sufficient to remove all artifact effects, and no explanation was provided for artifacts produced by RMA.
In the present paper, we extend the analysis performed by , aiming to shed more light on the behaviour of multi-array techniques specifically in the context of inter-array correlation. We will describe the characteristics of probesets which induce these artifacts and provide both a mathematical and a biological explanation for the phenomenon. Finally, we introduce a slightly changed version of the RMA code which massively reduces inter-array correlation artifacts, while retaining RMA features in the context of differential gene expression analysis.
average values for inter-array correlation coefficients
MAS5 alone shows the expected no-correlation behaviour. It must be noted that, unlike the other two techniques, MAS5 uses a single-array summarization technique (a robust Tukey-biweight average of the probe values) which treats each sample separately.
We will focus on the cause of this behaviour observed when using RMA and GCRMA, trying to understand the mathematical and biological scenarios that could introduce such a massive artificial inter-array correlation for these two methods.
Returning to our original Arabidopsis dataset, we observed that many probesets seem to yield completely identical values across different samples when processed by RMA or GCRMA. Datasets of three arrays normalized by RMA and GCRMA show, respectively, around 20% and 12% of the probesets population with identical values across all samples. The effect will decrease with increasing sample size (see Figure 1B) as previously reported in . We therefore measured the tendency to yield identical expression estimates for any particular probeset after RMA normalization (ID tendency, see Materials and Methods) and compared it to several probeset characteristics.
This phenomenon is also particularly evident for lowly expressed probesets (Additional file 3, Figure S6) and those hybridizing to multiple targets (Additional file 4, Figure S7), especially if the different targets fall into different biological classes (Additional file 5, Figure S8).
In summary, RMA and GCRMA tend to yield identical values for probesets containing probes that yield grossly different measurements across samples, and therefore are either noise-driven or have multiple independent targets.
As the problem of bad probesets has been discussed before and been tackled by providing updated probeset definitions in the customCDF project , we assessed whether the oscillating behaviour for real data was still observable when using such an updated definition. However, qualitatively identical results were still obtained using such an updated probeset annotation (Additional file 6, Figure S1). This might be explained by the fact that expression is also inversely correlated to a probesets' tendency to give identical values across arrays.
Taken together, these results tell us that RMA (and GCRMA) introduce artificial correlation across microarrays driven by lowly expressed, internally inconsistent, multi-target and/or multi-function probesets.
To identify why this artifact arises during the median polish procedure, we investigated the algorithm further. RMA and GCRMA apply median polish by creating a matrix from the measured values within each probeset, placing probes along each row, and samples along each column. The medians are subtracted from the intensities to cumulate residuals in each step and the grand effect (or median of medians) is subtracted from medians to cumulate probe effects in each step (Additional file 8, Figure S10).
This algorithm is more likely to introduce identical values with odd and small sample sizes, like the one depicted in Additional file 8, Figure S10. In such a case, the row medians will fall on a specific value and be transformed to zero during the first row sweep (Additional file 8, Figure S10B, top panel), this will increase the chance to have a zero as column median during column sweep (Additional file 8, Figure S10C, top panel).
Overall, the RMA implementation of the median polish algorithm shrinks all values in the probeset matrix to similar or identical values, with a stronger effect for samples, since it starts subtracting probe (row) medians. In the example of Additional file 8, Figure S10, the final sample values will be calculated by adding the grand effect to each column effect, and will therefore be equal to 8 for all samples.
It could be argued that the median polish summarization step could be helpful in the context of Differential Gene Expression analysis, since it will flatten unclear probeset matrices and therefore highlight strong signals. However, the result of generating completely identical expression values across arrays is not always beneficial. Moreover, this effect can be dramatically reduced by swapping the order of row/column median subtraction within median polish, or equivalently, by transposing the matrix created for each probeset, placing samples on rows and probes on columns. This alteration will introduce a presumably harmless similarity between probes within a probeset (which are assumed to be measuring the same quantity, and which don't form part of the output) while massively reducing the artificial sample identity.
To confirm this, we re-implemented the median polish summarization by inverting the order of the two sweep steps (Additional file 8, Figure S10), in what we call "transposed RMA" or tRMA. As shown in Figure 4, the inversion of median subtraction steps alone reduces the median polish effect to a very small residual inter-array correlation. This can be explained by the fact that the likelihood for the sample effects to give a zero value in tRMA is very low during the first iteration, as it would require perfectly identical medians of raw probe values (Additional file 8, Figure S10). Effectively, tRMA transfers the artifact of inter-correlation between sample to an inter-correlation between probe (in a common probeset) effects, which might be more plausible biologically (as all probes in a probeset should measure the same target) and remains contained within the procedure and not yielded as output of the preprocessing method.
The inter-array artificial correlation effect introduced by the median polish step is increased in GCRMA (Figure 4, dark green dotted line). As previously discussed by , GCRMA contains a potential problem in its background correction step, that adjusts probe intensity values through gene-specific binding. This introduces artificial inter-array correlations between probes with similar binding affinity, and therefore strengthens the effect of the following summarization step. However, substituting the median polish step with our transposed alternative "tGCRMA" (Figure 4, dark green line), massively reduces the inter-array correlation between permutated samples.
comparison between RMA and tRMA in the affycomp benchmark
Signal detect slope
Signal detect R2
null log-fc IQR
null log-fc 99.9%
weighted avg AUC
The differences between RMA and tRMA are minor when compared to the results for an independent method (MAS5): tRMA shares most of the qualities of RMA, without introducing inter-array correlations. It is interesting to highlight the fact that tRMA yields a higher median standard deviation (Median SD, in bold in Table 2) between spike-in replicates. This effect can be wrongly interpreted as tRMA's lower sensitivity; however, we now know that the original RMA median polish implementation is introducing identical values across experiments, and therefore artificially reducing the variance between spike-in replicates as well.
Since median polish alters inter-array correlation, sample classification is a common analysis that could be affected by this summarization step. Thus, we analyzed the AtGenExpress stress dataset for Arabidopsis , and calculated the capability of both preprocessing techniques to separate roots and shoots samples (see Material and Methods).
In order to compare the relative performance of RMA and tRMA when filtering on differentially expressed genes, we used a dataset that was previously used by , to tune classification where the origin of the RNA in each sample was known. Choosing a sample size of 5 where 2 pairs of 2 samples each came from the same specimen and one sample came from a different specimen, tRMA yields better classification results for almost all FDR corrected p-value thresholds (Additional file 10, Figure S5).
However, when filtering out lowly expressed genes , RMA performed generally as well as tRMA when performing sample classification on this dataset (data not shown). Unlike for the tissue sample dataset, RMA performed better than tRMA when filtering based on variance in this use case even when adding random noise to all samples (Additional file 11, Figure S11A). Only when noise was selectively added to non-paired samples did tRMA outperform RMA (Additional file 11, Figure S11B).
Furthermore, RMA performs slightly better than tRMA in 7 out of 14 of the Affycomp tests shown. Although objectively minor, these differences point out that tRMA may not necessarily be an improvement over RMA in all types of analyses.
The use of GCRMA and RMA preprocessing algorithms for Affymetrix GeneChip technology has received a remarkably broad adoption in the community due to their low computation time and to their superiority with respect to other methods in previous benchmarks.
However, one of the most relevant advantage of RMA and GCRMA in the AffycompII challenge , the low variance across replicates, seems to be partially the result of artificial inter-array correlation. Extending what was already noted by , we show that the artificially high similarity between samples given by RMA and GCRMA is caused by the shared median polish summarization step. This artificial behaviour is particularly strong in internally inconsistent, noise-driven and multi-hit probesets, and as a consequence identical results across arrays are generated. We analyzed this artifact effect for the Arabidopsis thaliana ATH1 Affymetrix GeneChip, but we found highly similar results in exploratory experiments on other organisms and platforms (specifically, human HG133 and E. coli Asv2 - data not shown).
The median polish step doesn't seem to pose a particular problem in differential gene expression analyses, because, on the contrary, it could enhance the differences of changing transcripts by shrinking most unclear probesets to identical values across experiments. In any case, the small underlying change in gene expression of such an unclear probeset would generally be below the cut-off value to be considered an 'interesting' gene. It is interesting to note, however, that median polish has already been shown to work poorly when compared to MAS5 summarization in correlation between E. coli operon members .
However, this artificial correlation can't be ignored in contexts where unbiased measurements are needed, like transcript clustering , genetic network reverse-engineering , sample classification [22, 34] or global transcript models , and we show in the present paper how RMA can artificially decrease the gap between samples coming from different tissue types. Furthermore, we could show that filtering of differentially expressed genes leads to a worse sample classification performance for small odd sample sizes samples for RMA, when compared to tRMA. Nevertheless, depending on the use case, RMA performed better when filtering based on variance. Thus, our results raise issues, especially when small sample sizes are used, on the validity of many studies obtained on the basis of correlation measures after these normalization procedures were applied.
We propose a minor change to the median polish algorithm that will almost eliminate the correlation artifacts without significantly affecting any of the positive RMA/GCRMA qualities. We provide a modified version of RMA named tRMA as a standalone R package (additional files 12 and 13). The method contains a modified version of the BioConductor  preprocessCore and affy packages that alter the median polish summarization step as described in Additional file 8, Figure S10 (bottom panel). Our tRMA method will offer the possibility to use an unbiased normalization technique both for differential gene expression analyses and for correlative studies based on microarray data.
In order to obtain a vast, robust and condition-independent dataset, we downloaded all Arabidopsis thaliana ATH1 microarrays available from GEO  and removed truncated or unreadable files and genomic DNA experiments. This dataset comprised 3707 arrays and is henceforth referred to as the "Arabidopsis dataset".
To test the abilities of RMA and tRMA to correctly cluster different tissue samples, we analyzed microarrays from the AtGenExpress stress study , contained in the Gene Expression Omnibus series GSE5620-GSE5628. This dataset (root-shoot dataset) comprises 248 samples, evenly distributed in shoot and root tissues.
To further assess sample classification performance of RMA and tRMA, we focused on a human breast cancer dataset published by  and reanalyzed by . This dataset contains 98 surgical specimens, 18 of which belong to 9 replicate pairs in which two samples were taken from adjacent sections of the same frozen block.
In order to compare real samples with completely uninformative samples, we decided to randomly permute the raw signal intensities of the Arabidopsis dataset as in . In brief, every Perfect Match (PM) probe and its Mismatch (MM) counterpart were reassigned to a random probeset within the same microarray. This generates information-less probesets while keeping the properties of the original probe intensity distribution.
We compared the microarray preprocessing procedures RMA , GCRMA  and MAS5  using the software implementations available from BioConductor. In every case, the default parameters were used. All final outputs, including MAS5 ones, were analyzed on the log2 scale.
The behaviour of the three microarray preprocessing procedures was analyzed in the context of randomly selected subsets of the Arabidopsis dataset. Different sample sizes were selected (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 30, 50 and 100) according to the realistic scale of a single-experiment dataset. For each sample size, 1000 subsets were selected and normalized. For each normalized subset, we calculated inter-array Spearman correlations and then plotted the overall mean and standard deviation of these correlations for each sample size.
The same procedure was then repeated for the permutated Arabidopsis dataset.
where I is the final probe intensity, O and P are, respectively, the original intensity and a permuted intensity. w o and w p are the weights given to both (where w o +w p = 1). W p is referred to as noise level.
The model tries to predict every ith probe intensity rank in the jth sample (p ij ) using as explanatory variable the jth sample effect, S j , calculated as the probe's mean rank for the sample. The model will then try to adjust the sample effect weight w and the intercept i to minimize the unexplained error ε.
It is apparent that the R2 for this model will be high when all probes within a probeset behave consistently relative to each other across different experiments, i.e. when the probe rank in a specific experiment is predicted quite well by the probe's mean rank across experiments. On the other hand, a low R2 will result from probes acting inconsistently across experiments, e.g. with some probes ranking particularly highly in some experiments yet poorly in others. The modelling procedure is provided as an R function (fitLM, Additional file 14) to determine internal consistency of probesets together with a function to reproduce the figures (Additional File 15).
With the goal of reducing inter-array correlation artifacts without losing the positive features of RMA, we modified the RMA median polish source code of the preprocessCore library available on BioConductor . Our method simply changes the order of median substitution, starting from column (sample-wise medians) instead of from rows (probe-wise medians), and was therefore called "transposed RMA" (or tRMA). tRMA code is available in the Additional files 12 and 13 and can be run in the R environment .
In order to evaluate and benchmark our newly proposed preprocessing method, tRMA, we adopted the criteria developed for the AffycompII challenge [9, 6] using the two Affymetrix spike-in datasets HGU95 and HGU133 and the Affycomp online tool .
From the root-shoot dataset, we randomly selected 10000 groups of 5 arrays composed of 3 samples from one tissue type, and 2 from the other. Each dataset was normalized using tRMA or RMA and distances between arrays coming from the same tissue (intra-tissue distance) and between arrays coming from different tissues (inter-tissue distance) were determined. Distances were calculated as (1-Spearman correlation coefficient) using either all probe sets or only the 50% showing the highest variance.
Secondly, a dataset previously used by  to assess microarray performance was used to determine the percentage of correctly clustered subsets of 5 microarrays. From the dataset, two couples of samples coming from the same tumor or non tumor specimen, plus a different specimen were sampled. Probe-sets were selected based on differential expression between the samples using the limma package applying different p-value thresholds corrected using the Benjamini-Hochberg method . These p-values are obtained by a specific combination of empirical Bayes methods and linear models described in . The outcome of the normalization was defined as "correct" if, for every sample in a couple, its highest correlation coefficient against all other samples is the other correct member of the couple, which would lead to them being clustered together. The sampling was repeated 1000 times for each different p-value. The increase in the performance of tRMA when compared to RMA was assessed using a Fisher's exact test with Benjamini Hochberg correction.
The human dataset was used also to perform a test on clustering perfomance on groups of genes sorted by variance, as described by , but using only subsets of five samples (belonging to three groups). This test was performed for RMA and tRMA at different probe noise levels, added following the procedure described previously in the "Noise robustness analysis" paragraph. The noise was either added to all samples uniformly (Additional file 11, Figure S11A) or only to two unrelated samples (i.e. belonging to different replicate groups) (Additional file 11, Figure S11B).
Robust Multiarray Algorithm
transposed Robust Multiarray Algorithm
MicroArray Suite 5
Perfect Match probe
We thank Dirk Steinhauser for insightful comments on the draft and Andrea Califano and Wei Keat Lim from Columbia University for providing the microarray permutation algorithm.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.