Identity scoring scheme

The basic relevance network algorithm (scoring with unit matrix) is used to compare the performance of all measure's classes.

Measures operating on random variables

Furthermore, the ID scoring scheme is evaluated using several measures which employ time series represented via random variables. In particular, we examine in detail the performance of correlation and information-theoretic measures.

In the case of the linear Pearson correlation (PC) coefficient, as shown in Figure 2(a), we obtain almost identical results from the simple and the conditional (CPC) measure, although the CPC is expected to eliminate indirect interactions. However, this does not mean that there are no indirect links wrongly deduced by the linear PC. The problem here is rooted in the estimation of the conditional probabilities, which is barely reliable for 10 time points.

Even if a basic significance test is included -- for example the data is reshuffled 100 times, then the measures for the randomized series are calculated, and the results are compared to those obtained from the original time series -- the results do not change significantly (Additional file 1, Figure S2). The partial Pearson correlation, on the other hand, shows better results for low false positive rates, but looses its accuracy when high true positive rates are reached. Additionally, the results obtained from the PPC are less significant (in terms of the reshuffled time series). Removing links which have no significant values of the correlation leads to an almost random prediction from the partial Pearson correlation.

As we cannot infer self-regulation by analyzing the similarity of expression series, the diagonal of the correlation matrix was set to zero in our computations above (by definition it is one). Comparing the reconstruction efficiency of the linear PC with that of the rank correlations (diagonal equals to one), we observe that the ROC curve shown in Figure 2(b) is smoother for the Pearson correlation than the curves obtained from the rank correlations. Hence Pearson's correlation measure is less sensitive to the choice of the threshold, whereas the rank correlations can achieve a slightly better overall performance.

Next, we investigate the efficiency of the ID scoring scheme considering information-theoretic measures. In general, we observe that the resulting reconstruction strongly depends on the method chosen for the estimation of entropies. Here we present the results obtained using the R-package "infotheo" (in particular the Miller-Madow asymptotic bias corrected empirical estimator) since, for short time series, it yields better estimates of the entropy than the R-package "entropy". Besides the basic pairwise mutual information (MI), we also investigated the conditional mutual information (CMI) and the residual mutual information (RMI) in order to reduce the number of false positive links. All these measures result in ROC curves which are more or less discontinuous. This is a finite size effect, as the time series are very short, and thus the estimation of the MI (entropies) becomes problematic.

We find a quite different behavior of the ROC curves, as shown in Figure 2(c), in specific regions of the ROC space. The simple mutual information results in a flat and comparatively smooth ROC curve for high false positive rates. This means that the measure allows removing about 60% of the false positives, by loosing approximately 10% of the true links. An even better performance in the same ROC space region can be achieved using the residual mutual information, which we proposed as a partial mutual information measure to distinguish indirect from direct (linear) relationships between triplets of genes. In contrast to this, the conditional MI results in a more discontinuous curve for high fpr: here, the ratio of the true and false positive rate is nearly the same as observed for a random prediction. In principle, the CMI is stricter in removing indirect links as it also can detect nonlinear interactions. However, the conditional probabilities cannot be estimated sufficiently well from 10 time points. Hence, the conditional MI fails for (the investigated) short data sets in the region of high false positive rates.

Additionally, when looking at the region of low fpr, we observe that the ROC curve of the simple MI becomes more discontinuous than for the high fpr. The true positive rate decreases significantly for slightly reduced threshold values, in the region around 30% and 15% of the false positives. This is manifested as jumps in the curve due to which this measure is rather sensitive to the choice of the threshold if low false positive rates are to be achieved. In contrast to this, the residual mutual information results in a smoother curve for low false positive rates than the simple MI, indicating that the measure is less sensitive to the choice of threshold, although the curve exhibits smaller jumps as well. In the region of fpr < 10% the performance of the RMI decreases slightly compared to the simple measure. The conditional mutual information on the other hand, achieves only very low false positive rates, which also lead to low true positive rates (up to about 5%). Tuning the threshold to allow for slightly higher values of the fpr the ROC curve of the CMI immediately jumps to 50% of false positives. Hence, the region between about 3% and 50% of false positive links is not achievable using the considered conditional measure.

We also implemented a basic significance test for the mutual information measures by reshuffling the time series 100 times, calculating the measure for the randomized series, and comparing the results to those obtained for the original time series. The associated ROC curves are shown in Additional file 1, Figure S3. With respect to the significance, the reconstruction efficiency of the simple and, in particular, the residual mutual information decreases, since the inferred degree of interaction for most of the gene pairs is not significant in the specified sense. In contrast to this, with the significance test, the quality of the prediction obtained from the CMI increases slightly, but its overall performance is still deficient.

That evaluation leads to the conclusion that (from the MI measures) only the simple and the residual mutual information can provide a sufficient reconstruction efficiency using the IDentity scoring scheme. This holds true only in the case that we do not rely on the simple significance test.

Investigating the performance of the coarse-grained measures on the short gene expression time series, we obtained ROC curves which look almost the same as expected for a complete random linking in the network, as illustrated in Additional file 1, Figure S4. Even though the coarse-grained measures are in principle promising for the inference of interdependency from time series of intermediate length, they are not applicable in our case. The reason for this is the limited number of available time points which makes not only the estimation of the MI, but also the identification of a proper time lag a very challenging task. Interpreting the CCIR as a distance, and not as a similarity measure, leads to an increase of the inferred true positives. However, the predictive power of the measure remains very low.

Symmetric scoring schemes - CLR, ARACNE and MRNET

Next, we evaluate the possibility for reconstruction of the underlying E. coli (sub)network based on the three modifications of the relevance network algorithm (Algorithm 1, given in the Methods section) as implemented in the "minet"-package, namely the CLR, the ARCANE and the MRNET. All three algorithms represent extensions of the basic relevance network approach, to the effect that they introduce additional scoring rules for the pairwise weighting of the interactions in order to reduce the amount of links that are falsely detected.

In Additional file 1, Figure S6, we present the results, in terms of the ROC curves, which we obtain using the different scoring schemes and in all cases the default weights of the pairwise interactions, namely, the squared Spearman's correlation for every set of pairs. As the algorithms implemented in the "minet" are designed to reduce the number of false positives, high false positive rates (of more than about 50%) do not occur here, unless all interactions are set as links. Moreover, the MRNET and the CLR result in ROC curves which are not smooth, meaning that their capability to reconstruct particular links is limited and strongly dependent on a proper choice of the threshold τ. The ARACNE, on the other hand, is restricted to an almost fixed fpr-tpr value.

Asymmetric scoring schemes - TS and AWE

As none of the previously described scoring schemes is able to indicate directionality from symmetric measures, we include in this study, and to our knowledge for the first time in GRN reconstruction, an evaluation of the performance of the Time Shift (TS) as a symmetry-breaking scoring scheme. We show the results of this modification of the relevance network algorithm removing the links which are falsely detected by the CLR (measure: μ _ρ ) or the AWE (measure: ). However, unraveling the directionality of interaction (between pairs of genes) using the correlation of the delayed time series has shown to decrease the maximal achievable true positive rates.

The slope of the ROC curves (shown in Figure 2(e)) indeed does not change much in comparison to the results of the CLR and the AWE scoring scheme. Moreover, if combined with the Pearson correlation of the delayed time series, the ROC curve obtained from the CLR becomes considerably smoother (compared to the curve shown in Additional file 1, Figure S6) and hence, the prediction is less sensitive to the choice of a threshold. The same does not hold true for the ROC curve obtained from the application of the TS scoring scheme in addition to the AWE. Instead, this curve becomes flatter and is slightly shifted towards lower false positive rates in comparison to the corresponding curve in Figure 2(f). This implies that, while for low fpr the curve looks basicly the same, in the intermediate range of the ROC space (fpr about 0.15 to 0.45) similar tpr values can be obtained for lower fpr. However, in the range of high fpr, the maximal achievable tpr value is lower. Hence, true positive rates of approximately 80% can be achieved with lower costs, as the according number of false positives is in general smaller when the TS scoring scheme is used. On the other hand, as already mentioned, the quality of the link detection becomes worse for higher false positive rates (more than about 40%) compared to the corresponding results of the AWE itself. The true positive rate in the ROC curve in Figure 2(e) is almost constant in this region of the ROC space.

Similar to the TS, the AWE scoring scheme aims at breaking symmetries and thus allows extraction of information about the directionality of interaction from symmetric measures. However, a detailed comparison of the reconstruction efficiency of the AWE using different symbolic dynamics measures shows that in contrast to the TS scoring scheme, AWE does not decrease the maximal achievable true positive rates. Instead, the ROC curves, as shown in Figure 2(f), become more flat for high false positive rates compared to the curves obtained for the basic algorithm with the ID scoring scheme using symbolic dynamics (Figure 2(d)). Hence true positive rates of more than 80% are achievable by the AWE algorithm with much lower costs than with the ID scoring scheme. On the other hand, the ROC curves obtained from AWE are more steep for low false positive rates. This implies that here true positive rates up to approximately 45% can be achieved with false positive rates of less than 10%. Furthermore, the curves shown in Figure 2(f) are much smoother in comparison to those in Figure 2(d), indicating that the reconstruction is less sensitive to the choice of a particular threshold.

Influence of noise

In general, noise-free expression measurements cannot be achieved in real experiments: In fact, intermediate and high noise level are not rare. Thus, in order to account for stochasticity in the time series, and particularly to establish the robustness of the ranking of the investigated similarity measures, we additionally evaluate the ROC curves for noise intensity of 0.3.

As expected, the measures which failed in the noise-free case (e.g., DTW, CMI, the coarse-grained information rate, and the Granger causality measures) did not improve their performance, as shown in Additional file 1 Figure S7. On the other hand, the measures based on vectors yield very robust results with respect to noise (Additional file 1, Figure S7). However, since the performance of these measures was already insufficient in the noise-free case, its general overall ranking does not improve significantly. We additionally noticed that the measures which performed best in the case of optimal sampling conditions, such as MI, RMI, correlation and symbol based measures differ in their robustness against noise, as illustrated in Figure 3. For example, the reconstruction efficiency of the simple and the conditional Pearson's correlation slightly decreases, while that of partial Pearson's correlation slightly increases. Hence, all three measures result basically in the same ROC curves, meaning one can abandon the more computationally intensive calculation of partial and conditional Pearson's correlation under these circumstances. Furthermore, MI and RMI both lose their accuracy as noise increases, and the corresponding ROC curves resemble those of the Pearson's correlation. However, the relation between both measures stays the same (RMI performs slightly better than MI).

The reconstruction efficiency for the symbol based measures decreases significantly as well, which holds true in particular for the mutual information of symbol sequences (as noise affects the inference of a symbol sequence using order pattern, as well as the binning process for MI calculation). However, apart from that, the ROC curves obtained for the symbol based measures are more continuous for noisy data than those in the noise-free case, which implies that the reconstruction process in this case is less influenced by the choice of a particular threshold.

A similar behavior is observed for the rank correlation coefficients. However, the shape of the curves appears more robust under the influence of noise than it is the case for the symbol based measures. Hence, the rank based measures represent the most suitable similarity measures to study the interrelation among short time series at high noise levels.

Finally, we observe that the CLR and the AWE are the most robust scoring schemes with respect to noise, whereas ARACNE fails for short and noisy time series.

A detailed analysis on the reconstruction efficiency of the top-ranking measures and scoring schemes under various stochastic conditions is considered in the Discussion section. Additionally, the performance as a function of the length of the time series and the noise intensity can be found in the Additional files.

The role of interpolation and sampling

Due to the fact that time-resolved gene expression data are usually quite coarsely sampled, general assumptions upon what happens between two time points cannot be made. This problem becomes obvious when unequally sampled data are used (Additional file 1, Figure S8).

Although the interpolation at the beginning of the time series (where the time points are rather close) seems to be sufficient, it does not accurately capture the dynamics of the expression time series when the distance between the time points becomes larger. Hence, by interpolating the gene expression data sets, artifacts are introduced, which will be further reflected in the results of the particular measures of interdependency. In order to avoid these artifacts, we renounce the interpolation in this comparison study, even though this leads to less significant results for almost all measures, as they operate far below the limit of their theoretically defined preconditions. However, we have observed that the overall results (ROC analysis) are typically equal or even better when interpolation is not included, especially when non-uniformly sampled time series are considered. Additional file 1, Figure S9 illustrates this effect exemplary for the simple mutual information.

However, some measures, such as the Granger causality used in this study, as well as several scoring schemes (e.g., the Time Shift), are explicitly time dependent. Hence, they require uniformly sampled data, meaning that an interpolation is needed if only non-uniformly sampled data is available. This is in general, the case in GRN reconstruction.

However, most of the well performing reconstruction tools in our study are not explicitly time-dependent, which means they do not require a specific time sampling. This implicates that they are not very sensitive concerning the spacing on the time axis. Our results, as shown in Additional file 1, Figure S10, illustrate that a non-uniform sampling for these tools can even improve the quality of the reconstruction, since a larger period of the dynamics is captured.

The role of the network topology

In general, the underlying network and its properties are not known prior to the reconstruction process. However, the available experimental and theoretical research has suggested that gene regulatory networks most likely are characterized with scale-free properties [39]. Therefore, we compare the reconstruction efficiency of the relevance network approach for various subnetworks of E. coli's and S.cerevisiae's regulatory networks, as described in details in the Methods section (subsection on Synthetic data sets). These networks (Additional file 1, Figure S14) differ in size, average degree and clustering coefficient. Nevertheless, we observe that the performance of the top-ranking measures, such as the symbol based measures, rank correlations, MI and RMI do not depend on the network topology: Very similar ROC curves are obtained for all of the network types analyzed, as shown in Figure 4 (this also pertains for several other measures, as shown in Additional file 1, Figure S11). The performance in the range of low fpr was improved for most of the measures for increased average degree of the nodes. However, at the same time, the performance in the range of high fpr was usually decreased. In general, the largest differences in the reconstruction efficiency occur for the conditional Granger causality and partial Pearson correlation, where the quality of the reconstruction decreases significantly for an increased number of nodes (e.g., E. coli network with 200 nodes) and an increased clustering coefficient, as in the S.cerevisae network.

ROC statistics for noise-free data

AUC(ROC) > 0.8,

YOUDEN > 0.5 and

AUC(PvsR) > 0.05,

0.8 >AUC(ROC) > 0.7,

0.5 >YOUDEN > 0.4 and

0.05 >AUC(PvsR) > 0.03

By calculating the summary statistics in the noise-free case, as shown in Figure 5, we conclude that several information-theoretic measures (simple and residual MI), correlations (simple and conditional Pearson's, as well as Spearman's) and measures based on symbolic dynamics (SyMI and SySimMI) perform sufficiently well in combination with the basic relevance network algorithm (IDentity scoring scheme). Here, it stands out that the simple Spearman correlation performs better than the simple Pearson correlation, and SyMI is better than the simple MI. This is due to the fact that symbol and rank based measures are less sensitive to finite size effects and the distribution of data.

The modifications of the relevance network algorithm in the "minet" package having best performance in the reconstruction of GRN from short data sets, are the CLR and the MRNET ("minet" is based on Spearman's correlation in this case). Here the AUC(ROC) indicates almost no change compared to the basic algorithm with identity scoring (measure: Spearman's correlation), while the Y OUDEN index decreases for the CLR and increases for the MRNET. However, the opposite is true for the AUC(PvsR). The overall performance of the CLR (in terms of the considered summary statistics) is slightly better than those of the MRNET (CLR scoring scheme was used to set the benchmarks). Moreover, the measures combined with the TS scoring scheme perform sufficiently well. However, the summary statistics do not change much compared to the results obtained for the same measures using the ID. In contrast, the asymmetric weighting yields a significant increase among all the summary statistics compared to the performance of the same measures using only the identity scoring scheme.

The asymmetric weighting (AWE) significantly improves the prediction at this point, since it breaks the symmetry of a particular measure based on topological consideration and, therefore, reduces the number of false positive links. Hence the AWE (measure: ) clearly shows the best performance when short time series are considered (the results become slightly better if Time Shift is applied in addition).

ROC statistics for noisy data

In order to account for stochasticity in the time series as well as to establish the robustness of the investigated (top-ranked) similarity measures against noise, we evaluate additionally their performance for two different noise intensities, namely 0.3 (Figure 6) and 0.5 (Additional file 1, Figure S12). Only those measures which perform sufficiently well in the noise-free case (measures operating on random variables and symbolic dynamics) are tested. In particular, we examine the Pearson's (μ _P (x, y)), Spearman's (μ _S (x, y)) and Kendall's (μ _K (x, y)) correlation coefficients as well as the symbol based measures , , , and using the ID scoring scheme. Additionally, we investigate the performance of CLR, MRNET, ARACNE, AWE and TS scoring schemes based on the same measures as in the noise-free case.

Under the influence of noise, the quality of the results of the symbol based measures (in particular ) decreases. As noise strongly influences the process of symbol assigning, it can principally enhance or distort the information content. The direction of the influence is not predictable a priori, but in the presence of strong noise, symbols are no longer reliable (if no additional information on the influence of the noise is provided). On the other hand, measures operating on random variables are rather robust against noise (the best results in these cases have been achieved using rank correlations).

The ARACNE has proven to be very sensitive with respect to noise. In contrast to this, the asymmetric weighting (compared to the results of the ID using the same measure) still performs well within the given limits, as it is only based on topological considerations, and it is not influenced by the presence of noise. Furthermore, to investigate how noise influences the reconstruction efficiency, we calculated the area under the ROC curve and the YOUDEN index as a function of the noise intensity for the 5 combinations of similarity measures and scoring schemes which performed best in the noise-free case, namely the symbolic measures and the asymmetric scoring schemes, mentioned in the previous section. Additionally, we compared the results to those obtained for time series of different lengths (i.e., 8 and 20 time points). We conclude that for short time series, the capability of the measures and scoring schemes to detect true and at the same time eliminate false positive links depends both on the number of time points and the noise intensity (Additional file 1, Figure S13). However, this dependence is small compared to the differences in the reconstruction efficiency between the various measures. Moreover, the sensitivity against noise is reduced with increased length of the time series (which corresponds to the usage of order pattern of higher dimension). In general, we observe a decrease in the reconstruction efficiency if the noise levels increase or the length of the time series decreases. For the short time series used in this study, however, these dependencies are not monotone.

Synthetic data sets

The evaluation of the existing methods for reverse engineering gene regulatory networks often employs real time-resolved expression data. However, these data include the convoluted effects of regulons (genes under regulation by the same regulatory protein) and stimulons (genes under regulation by the same external influence), which renders it challenging to realistically assess the performance of investigated methods. Moreover, not every regulatory subnetwork leads to expression of the participating genes over the measured time period and particular condition of interest. These facts lead to a lack of control when using transcriptomics time series data sets for network inference.

Following the example of the DREAM challenge [2, 14], in this comparison study we use synthetically generated data sets to overcome the described disadvantages. The usage of these synthetic data, in contrast to real measurements, enables us to directly compare the performance of different reconstruction tools, since the topology and dynamic of the underlying network is known a priori. In particular, we use the synthetic network generator, SynTReN [41, 42], which creates synthetic transcriptional regulatory networks by providing the edges of the network, as well as information about the interaction (activating, repressing, or dual). Additionally, the generator produces simulated gene expression data of the associated mRNA concentrations for each gene, based on Michaelis-Menten and Hill kinetics, which approximates experimental expression measurements. These expression data sets are uniformly sampled in time.

In SynTReN, the levels for three types of noise are user definable: (1) biological noise, corresponding to biological variability given by the stochastic variations in gene expression, (2) experimental noise, corresponding to the technical variability, and (3) noise on correlated inputs, which accounts for the influence of several activated genes on a regulated gene. Note that different noise levels are included in this study, but we make no distinction between the strength of the three noise types.

In particular, we generate regulatory networks from the GRN of E. coli and S. cerevisiae, using the cluster addition strategy to select a connected subgraph: In each iteration, a node is randomly chosen and added to the graph together with all its neighbors. This strategy is chosen since it is an efficient method to extract a subnetwork that approximates well the topology of the source network. The results presented in this work are obtained for subnetworks of distinct sizes which differ in degree and clustering coefficient. In particular, we investigate an E. coli subnetwork of 100 genes including 121 links where 10 of the genes code for transcription factors. It is characterized by an average degree of 2.42 and a clustering coefficient of 0.016. Additionally, we examine two more networks: (1) An E. coli subnetwork of 200 genes (34 coding for transcription factors) that includes 303 links and is characterized by an average degree of 3.03 and a clustering coefficient of 0.019, and (2) a S. cerevisiae subnetwork of 100 genes (14 coding for transcription factors) that includes 123 links and is characterized by an average degree of 2.46 and a clustering coefficient of 0.026. The degree distributions are shown in Additional file 1, Figure S14. Synthetic gene expression data with 10 biological (n = 10 time points) and 6 technical (r = 6) replicates have been generated from the particular networks. We use the means of the technical replicates for the interaction analysis. An example of a generated network (100 in E. coli used for our investigations), and the simulated gene expression data sets are visualized in Figure 7.

Interpolation

Since transcriptomics time series data usually consist of expression at a few, possibly non-uniformly sampled time points, data interpolation is often the first pre-processing step. Different techniques, including linear [43] and nonlinear interpolation methods, such as cubic- [33] and b-splines [25] have been applied [11]. Although these methods extend the available data, they also introduce artefacts due to over-fitting. For this reasons, we preclude from interpolating the synthetic time series data (whenever possible) for ranking of the used similarity measures. Nevertheless, for reason of comprehensiveness, we consider the effect of interpolation using cubic splines on the investigated similarity measures and scoring schemes of highest rank in absence of noise.

Similarity measures

Reverse engineering of regulatory networks relies on the inference of interrelationships among genes, based on similarity measures. In general, given two time series x and y over n time points, a similarity measure is given by the mapping μ : R ⁿ × R ⁿ → I, where I, I ⊆ R. The so-defined pairwise similarity measure detects (non)linear relationships between two variables, in the considered case, between two gene expression time series. The definition allows for the measure to be symmetric, which is not commonly the case for gene regulatory interactions. Moreover, if two genes are linked indirectly via a third gene, the pairwise measure can not distinguish direct from the indirect relationships and hence additional false positive links will be introduced in the network reconstruction.

Nevertheless, the definition of the similarity measure can be extended to conditional and partial measures, incorporating the possibility to exclude the influence of a third gene. Conditional similarity measures are more general, since they do not rely on specific assumptions on the probability distribution (deduced from the time series associated with a discrete random variable), but estimate the distribution which in turn impedes the computation of the measure from short time series. On the other hand, partial measures can indicated conditional independence reliable only for multivariate Gaussian variables.

To be able to discern the direction of a putative interaction and hopefully eliminate any spurious effects, we consider the conditional and partial variants for several of the measures detailed below. We term the basic pairwise measures as simple, in comparison to their conditional and partial variants. An overview on the measures included in this study is given in Table 1.

Measures operating on vectors

Some of the standard measures used for determining gene regulatory interactions are based on the calculation of the distance between expression time series regarded as vectors. In the following, x and y will denote the vectors <x₁, ..., x _n > and <y₁, ..., y _n >, respectively. Our study includes:

L ^s norm:

This distance measure for vectors x and y is defined as follows:

(1)

In our study, s = 10, which corresponds to the number of available time points.

Euclidean distance

Furthermore, we consider the well-known Euclidian distance, which is a special case of the L ^s norm, with s = 2. Therefore, it is defined as

(2)

Manhattan distance

We also study the performance of the Manhattan distance which represents the shortest path between two points, placed on a rectangular grid, and is analogous to the L¹ norm:

(3)

Dynamic time warping (DTW)

In addition, we investigate the performance of the DTW, which to our knowledge, has not been applied to the problem of gene regulatory network inference, but rather on clustering genes expression data [43, 44]. The DTW-based measure relies on finding the optimal (least cumulative) distance mapping a given time series into a reference time series, where both sequences may vary in time and/or speed. It was originally developed for speech recognition [45, 46], but has been recently used for different data mining tasks in medicine and bioinformatics [43, 47]. The concept of DTW is sketched in Figure 8 for two short time series with 4 time points each. In the first step of the DTW algorithm, local distances (e.g., Euclidean or Manhattan distance) for all pairs of time points are calculated. Then, the time series are mapped into each other by linking various time points, such that each point is included at least once and the sum over the lengths of all those links is minimal (optimal alignment path). Here, we use the DTW as implemented in the R-package "d tw" [48–50], with the Euclidean as point-wise local distance, and different step patterns which indicate the local constraints of the alignment paths. We include three different step patterns, namely

• symmetric1

  

  (4)
• symmetric2

  

  (5)
• and asymmetric

  

  (6)

to find an optimal alignment. Here μ _EC denotes the local (Euclidean) distance, and the measure μ _W the cumulative distance (representing the minimum sum of local distances along the alignment paths).

The resulting matrix of cross-distances D contains the pairwise calculated distance measures (μ _EC , μ _L , μ _MA , or μ _W , as defined above) and is, in all cases, normalized by the largest value occurring in the matrix as follows:

(7)

The similarity measure is then defined by:

(8)

Measures operating on random variables

Despite the representation of the expression time series as vectors, time series x = <x₁, ..., x _n > can be associated with a discrete random variable X with probability distribution p(x), x ∈ X that is approximated by the frequency via standard binning arguments. This representation allows to calculate several widely used similarity measures, such as correlation and information-theoretic measures. Note that the temporal information is lost by this representation of time series data. We first review the most commonly used measures in detail:

Pearson correlation

This similarity measure quantifies the linear relationship between the random variables X and Y, corresponding to two time series x and y:

(9)

where E denotes expectation

(10)

If the variables are independent, the correlation coefficient is μ _p = 0, but the opposite is not true, as this coefficient is sensitive mainly to linear dependencies. Note that μ _P receives values in the interval [-1, 1] and is symmetric.

Conditional Pearson correlation (CPC)

By using the conditional expectation value

(11)

where p(x|y), x ∈ X, y ∈ Y is the conditional probability distribution, one can provide the following definition for CPC:

(12)

Thus, the conditional correlation among the time series x and y of the corresponding genes, eliminating the influence of all other genes is defined as

(13)

Partial Pearson correlation

Analogously, one could also consider

(14)

where

(15)

The residuals are calculated following Eq. (16) making a linear regression of x (respectively y) depending on z:

(16)

Rank correlations can be used as a more general measure of interdependence, not restricted to a linear relationship. Even though they measure a different type of relationship than the product moment correlation coefficient, like the previous correlation measures, these are also defined in the interval [-1, 1].

Spearman's rank correlation

This type of correlation is based on the rank distribution of the expression values:

(17)

where R(x) is the rank of x.

Kendall's rank correlation

Another rank correlation is

(18)

with n _c being the number of concordant pairs, and n _d the number of discordant pairs of the rank sets.

It is common to regard the rank correlation coefficients (especially Spearman's rank correlation) as alternatives to Pearson's coefficient, since they could either reduce the amount of calculation or make the coefficient less sensitive to non-normality of distributions. Nevertheless, they quantify different types of association.

Unlike most of the measures discussed here, the correlation measures do not only provide an information about whether two genes are interacting, but also whether it is an activating or repressing relationship. As the latter information is outside of the interest of the current study, only the absolute value (respectively the square) of the correlation coefficient is taken into account.

Additionally to the correlation, information-theoretic measures are also defined using random variables as relevant representation for expression time series.

Mutual information

One of the most commonly used measures for inferring interdependencies between two subunits of a system is the mutual information (MI) [16]. Intuitively, MI measures the information content that two random variables X and Y share. The simple mutual information can then be expressed in terms of the marginal entropies H(X) and H(Y ), and the joint entropy H(X, Y ) using the definition of the Shannon entropy

(19)

which quantifies the uncertainty associated with a random variable. Hence the simple MI is defined as

(20)

It includes also non-linear interrelations, but same as the other simple measures, the simple MI cannot be used to distinguish between direct and indirect relations.

Conditional mutual information

However, if we replace the marginal and joint entropies by the conditional analogs, H(X|Z), H(Y|Z) and H(X, Y|Z), we can eliminate the influence of a third variable Z. Hence, we calculate the minimal information shared by the time series x and y of two genes conditioned on each z _k , k = 1, ..., m. Thus the conditional--sometimes also referred to as partial [51]--mutual information (CMI) can be written as:

(21)

Where

(22)

The degree of interaction is indicated by the values of μ _I or normalized by the largest value occurring among all pairs of genes.

Mutual/conditional coarse-grained information rate

Another approach based on information-theoretic aspects are the coarse-grained measures. Here, instead of approximating the exact entropies of time series, relative measures of "information creation" are used to study the interrelationship of two (sub)systems. Thus, for this purpose, the calculation of coarse-grained entropy rates [52] is used to replace the approximation of the Kolmogorov-Sinai entropy (metric entropy of a dynamical system): First, a time lag l _max is determined such that

(23)

among all analyzed data sets. Then, the coarse-grained information rate (CIR) is given by the norm of the mutual information

(24)

Usually the parameter l _min and Δl (difference between consecutive time lags) can be set to one, and thus the CIR becomes

(25)

Hence, the mutual coarse-grained information rate (MCIR) is defined as

(26)

whereas the conditional coarse-grained information rate (CCIR) as

(27)

with

(28)

Finally, a normalization by the largest value occurring among all pairs of genes is performed. These (normalized) coarse-grained information rates are then used to indicate the degree of interaction.

Model-based measures

Granger causality

A rather new approach for inferring gene regulatory networks is the Granger causality (GC). Given the time series x and y, two linear autoregressive (AR) models are estimated, both including the past of x, and additionally, one of them including the past of y. In order to determine the optimal order q of the AR model, which denotes the number of past time points which have to be included, we use the function "VARselect" from the R-package "vars" [53, 54] based on the Akaike information criterion (AIC) [55]. The AIC is a measure of the goodness of a fit of an estimated statistical model, deduced as a tool for model selection. In the general case, the AIC is defined as:

(29)

where u is the number of parameters in the statistical model, and L is the maximized value of the likelihood function for the estimated model.

With properly selected AR models, the part of the variance in the data which is explained by one model in comparison to the other one, provides an information on the causal relationship. This comparison can be formulated in terms of an index.

Thus the Granger Causality index, denoted by μ _G for the simple linear measure, as defined in [34, 35] via the covariance σ, is:

(30)

and can be inferred from the AR models:

(31)

(32)

where a_11i, a_21iand a_22iare the parameters of the models and u_1t, respectively u_2trepresents white noise.

Conditional/partial Granger causality

As for the previous measures, we use the conditional and partial (linear) Granger causality measures (CGC and PGC) as defined in [34, 35], in order to identify existing indirect relationships. Hence, the AR models are formulated as

(33)

(34)

for the conditional, and, in addition,

(35)

(36)

for the partial Granger causality, with a_11i, a_12i, a_21i, a_22i, a_23i, a_31i, a_32i, a_41i, a_42iand a_43ibeing the parameter of the models and u_1t, u_2t, u_3tand u_4trepresenting noise terms.

Using the Eqs. (33) and (34), the conditional Granger causality index is then defined as:

(37)

where

(38)

and, using the Eqs. (33) to (36), the partial Granger causality index is

(39)

where

(40)

The degree of interaction is indicated by the Granger causality index normalized by the largest value occurring among all pairs of genes.

Measures operating on symbolic dynamics

Despite the promising applications of interaction measures based on symbolic dynamics in various fields, they have not yet been employed for reverse engineering gene regulatory networks. For instance, in standard nonlinear time series analysis, the usage of symbolic dynamics to uncover patterns of interactions, especially from short data sets [56], has proven as a valuable tool. Therefore, we explore the potential of symbolic dynamics for the problem at hand by using the principle of order patterns. By this principle, as described in [38], the time series are transformed into symbol sequences. An order pattern π of dimension δ is defined by the discrete order sequence of the time series x and has the length δ. Hence, the time series can be symbolized using order patterns following:

(41)

where l is the time lag. In terms of gene regulatory network reconstruction, we need to choose a certain number of time points and rank them according to their expression value in order to obtain the order pattern. Then, each possible ordering corresponds to a predefined symbol.

This concept is illustrated in Figure 9 for time series composed of n = 4 time points. As we are dealing with very short time series here, we consider all possible combinations of the chosen number of time points. For instance, for the time series of length n = 4 and an order pattern of dimension δ = 3, we define symbols (order patterns π _k ) for the following groups of time points: (1, 2, 3), (1, 2, 4), (1, 3, 4) and (2, 3, 4), shown in the left panels of Figure 9. Next, we define a symbol sequence

(42)

where denotes the order pattern obtained for gene i from the k-th group of time points and

(43)

is the length of the symbol sequence.

In this work, we usually choose the dimension δ such that the length T becomes maximal, given a time series of length n (i.e., δ = 5 for n = 10). However, if the symbol sequences are calculated for longer time series, this approach is not applicable anymore. This is due to the fact that calculations in R are not possible because of the fast growing length of the symbol vectors. Hence, we use for the time series of 20 time points order pattern of dimension 6 instead of the dimension 10, which would lead to the symbol vector of maximal length.

Symbol sequence similarity

Using the above described approach, we infer the interdependency of two genes as follows: Given a certain number δ of time points we define a vector P containing all possible permutations of the ranking, and assign a symbol (order pattern π _k ) to each of them. Next, we define a vector , using the same symbols as for P, but assigned to the reversed ranking. Now, we count the pattern overlap of two symbol sequences S⁽ⁱ⁾and S^(j)to evaluate the symbol sequence similarity, p₁, assuming both time series are interrelated (Eq. (44)), respectively p₂ if we assume anti-interrelation (Eq. (45)).

(44)

(45)

We choose the maximal value of the two frequencies p₁ and p₂

(46)

to define the symbol sequence similarity (SySim).

Mutual information of symbol vectors

We calculate the mutual information of the symbol vectors (SymMI) of maximal length by:

(47)

In addition, we consider the mean of the symbol sequence similarity and the mutual information (of the symbol vectors) (SymSimMI)

(48)

This is further extended to include symbolic dynamics based on a slope comparison (order patterns for pairs of time points), where we consider

• the symbol sequence similarity for pairs ( pairs) as a similarity measure

• and the conditional entropies for pairs ( pairs) as a distance measure, with .

A novel measure -- the residual mutual information

Estimating entropies from short time series is imprecise, hence the estimation of the mutual information and, in particular, its conditional counterpart, suffers the same disadvantage. On the other hand, the simple mutual information is not able to distinguish between direct and indirect links. Therefore, in order to overcome the encountered problem, we propose a novel partial measure -- the residual mutual information (RMI) defined as:

(49)

where

(50)

analogously to the idea of partial correlation (the residuals are calculated in the same way as for the partial correlation in Eq. (16)). The degree of interaction in the complex network is then indicated by the values of , normalized by the largest value occurring among all pairs of genes.

Applied to short data sets, we expect that the residual mutual information performs much better in discriminating indirect links than the conditional MI, as we can abandon the estimation of additional conditional probabilities. Hence, the measure is more robust to effects of small sample size. Furthermore, we expect that the RMI's performance ranges between those of the simple and the conditional mutual information for long time series, since in contrast to the CMI, we eliminate here only the linear influence of the variable Z on X and Y. We postpone the confirmation of this claim for further theoretic analysis.

Scoring schemes

Once a chosen similarity measure has been applied on a given data matrix, there are several possibilities to score the resulting "weights" of putative interactions. In this sense, a scoring scheme F is a matrix of dimensions m × m. Let W denote the matrix obtained by applying μ on all pairs of rows of a given data matrix M, for w _ij ≥ 0, ∀i, j. The scores from a given scoring scheme and similarity measure can then be represented by a matrix C calculated from the Hadamard element-wise product of W and F, such that c _ij = w _ij · f _ij .

To unravel the linkage of genes we apply the relevance network algorithm (Algorithm 1) using different scoring schemes and measures. In principle, all of the measures can be combined with any scoring schemes, but we restrict our investigations to the most commonly used.

Context Likelihood of Relatedness (CLR)

As a second type of scoring scheme, often used for the reconstruction of GRN, we consider the CLR, which is an extension to the basic relevance network approach. Once weights w _ij have been assigned for each pair of genes according to the strength of interaction inferred from a particular measure, a score is derived, related to the empirical distribution of the values in W. Thus, the matrix F obtains the form

(51)

(52)

(53)

where and σ _i (σ _j ) are the mean and standard deviation of the empirical distribution of w _ik (w _jk ), k = 1, ..., m. The links having c _ij <τ (with c _ij = w _ij · f _ij and τ a predefined threshold) are removed for the network reconstruction.

Here, we use the CLR as implemented in the R-package "minet"[57, 58], which uses either the simple mutual information or a squared correlation matrix (Pearson's, Spearman's or Kendall's) to measure the strength of interaction among genes. We note that the CLR algorithm cannot infer directionality from symmetric measures.

Algorithm for the Reconstruction of Accurate Cellular Networks (ARACNE)

Furthermore, we investigate the Algorithm for the Reconstruction of Accurate Cellular NEtworks, referred to as ARACNE, and include its performance in the current comparison study. The ARACNE is based on the data processing inequality, which states that post-processing cannot increase the amount of information. Hence it follows that:

(54)

when gene i and k are not directly linked, but this goes through j, where x _i , x _j and x _k are the expression time series of these genes. In this manner, the algorithm discriminates indirect links. ARACNE is a relevance network algorithm as illustrated in Algorithm 1 as well. First, weights w _ij (normalized to the interval [0, 1]) are assigned to each pair of nodes. Then the scoring scheme operates as follows: For each triplet of nodes the edge having the lowest weight will be removed (its weight is set to zero), if the difference between the two lowest weights is above a threshold τ _d , as that interaction is interpreted as indirect. In this manner, the matrix F obtains the form:

(55)

Moreover, the ARACNE removes all edges satisfying c _ij <τ, where τ is a predefined threshold.

As for the CLR algorithm, we rely on the "minet"-package, using the simple mutual information or a squared correlation matrix to determine the weights. By default, the two thresholds are set to zero. The ARACNE does not distinguish the direction of a link from a symmetric measure as well.

Maximum Relevance/minimum redundancy NETwork (MRNET)

As another example of a relevance network algorithm, we consider the Maximum Relevance/minimum redundancy NETwork (MRNET) [59]. This scoring scheme performs series of supervised maximum relevance/minimum redundancy (MRMR) gene selection procedures, where the expression of each gene in turn plays the role of the target output y = x _i , with V = x\x _i being the set of the expression data of the input variables, and x the set of the expression levels of all genes. Given the set M of selected variables and pairwise weights w _ij|M , the criterion updates M by choosing the variable

(56)

that maximizes the score

(57)

where is a redundancy term, and u _j = w _ji is a relevance term. This scheme therefore assigns higher rank to direct interactions, whereas indirect interactions (redundant information with the direct ones) should receive lower rank. Thus, the matrix F is defined as:

(58)

(59)

(60)

Finally, all edges whose score c _ij lies below a predefined threshold τ will be removed.

The implementation of the MRNET in the "minet"-package, which we used in our study, assigns the weights based on the pairwise simple mutual information or a squared correlation among the time series of two genes (normalized to the largest value occurring among the pairs). Also this algorithm is not able to infer directionality from symmetric measures.

Time Shift (TS)

In nonlinear time series analysis, the shifting of time series is a common way to infer the directionality of causal relationships. As the driving system by definition has to act first, shifting its time series forward in time (relative to the time series of the response system) should increase the similarity of both time series. Comparing the values of a particular measure for different time shifts gives then a hint on the direction of the interaction. Thus, the time shift scoring scheme starts with a cubic spline interpolation for each pair of genes expression time series. Then the series of the second gene is shifted against that of the first gene. If x and y are two expression time series stored in the i ^th and j ^th row of the data matrix M, and and are the related interpolated time series, we can then define the shifted time series as

(61)

and

(62)

where N is the length of the interpolated time series and N _shift is the assumed shift of versus , with N _shift ∈ Z and N _shift ∈ [-0.1 · N, 0.1 · N]. Next, is evaluated for all possible values of N _shift , resulting in a vector (for not significant values of μ the corresponding entry will be set equal 0). The scoring is now given by

(63)

(64)

(65)

In case the largest significant value of the measure is obtained for a negative shift, the regulatory direction from the first to the second gene is kept, while the opposite direction is preserved if the largest significant value is obtained for a positive shift. Furthermore, both regulatory directions are kept, if the maximum arises for a shift of zero or multiple opposed shift values or in the case when no significant value exists. The scoring scheme aims at providing a hint on the directionality, because the absolute values of the calculated correlations on the delayed time series are rather biased as the data sets are quite short.

In the next step of Algorithm 1, the information regarding the directionality are combined with the weight of interaction inferred from a particular measure (c _ij = w _ij · f _ij ). Hence, the weights for the unlikely direction are set to zero in order to break symmetries, and thus reduce the number of false positives. Finally, all edges with c _ij <τ are removed, where τ is a particular threshold.

We test that scoring scheme using the absolute value of of the correlation coefficients μ _P (Pearson) and μ _S (Spearman) for pairs of the shifted expression series, where the significance level was set to α = 0.01 and only absolute values of correlation larger 0.9 have been taken into account. The choice of the measure to infer the weights in the first step of Algorithm 1 is independent of that and includes here the mean of sequence similarity and mutual information of symbols, as well as Spearman's and Pearson's correlation. Furthermore, this scoring scheme is applied in addition to (or after) another scoring scheme (e.g., ID, CLR, or AWE). It is important to note that in contrast to the previously described modifications of the algorithm, the scoring scheme we propose here allows to investigate the directionality, also when symmetric measures are considered.

A novel scoring scheme - Asymmetric WEighting (AWE)

Most of the measures used to infer the degree of interaction between pairs of genes, such as correlations or the mutual information, are symmetric. Hence, when applied in symmetric algorithms they are not able to unravel the regulatory dependences, since these measures do not distinguish the direction of the interaction. Thus, we introduce an asymmetric weighting based on topological aspects, for the complete set of pairwise weights obtained from a particular measure, and implement it according to Algorithm 1. In particular, we compute a matrix of weights, where the columns represent the genes which are regulated, and the rows stand for the genes which regulate other genes. The scoring value is then calculated by dividing each row entry by the sum of the corresponding column values. The scoring scheme (and the corresponding matrix F) is defined by:

(66)

Hence, the probability that the j ^th gene is regulated sums up to one: . Here, the score indicates that how likely a gene is regulating another one depends not only on the strength of interactions, but also on its amount.

Eventually, if c _ij ≥ τ the edge is introduced, otherwise it is omitted.

We test the asymmetric weighting on the matrix W inferred from the symbolic dynamics measures.

ROC analysis

While discrete classifiers lead to just a single point in the ROC space, classifiers such as the similarity measures studied in this work produce probability values of how likely an instance belongs to a certain class. Here, the classification depends on a predefined threshold. The ROC curve is then produced by continuously tuning this threshold, which on the other hand can be suggestive on the performance of the measures. However, a well-defined rating is not always possible "by eye". Therefore, different summary statistics are common, for example the area under the ROC curve (AUC(ROC)) or the YOUDEN index (YOUDEN = max(tpr - fpr)) [61]. Another standard evaluation plot in the field of the ROC analysis is the precision/recall graph (PvsR), which is based on the comparison between the true edges and the inferred ones. Hence, it highlights the precision of the reconstruction, and does not suffer from the typically large number of false positives in a gene regulatory network reconstruction. We thus give the summary statistic using the area under the precision-recall curve (AUC(PvsR)) as well as the ROC curve. An efficient implementation of the ROC analysis is provided by the R-package "ROCR" [62].

All ROC curves are evaluated with respect to the underlying GRN, which is a directed graph. As several of the scoring schemes/measures do not distinguish whether the regulation is directed from gene i to gene j or vise versa, some of the false positives will follow from the missing information on the directionality. However, since the network under study is a sparse one, this additional false positives barely carry a weight.