The C1C2: A framework for simultaneous model selection and assessment

Background There has been recent concern regarding the inability of predictive modeling approaches to generalize to new data. Some of the problems can be attributed to improper methods for model selection and assessment. Here, we have addressed this issue by introducing a novel and general framework, the C1C2, for simultaneous model selection and assessment. The framework relies on a partitioning of the data in order to separate model choice from model assessment in terms of used data. Since the number of conceivable models in general is vast, it was also of interest to investigate the employment of two automatic search methods, a genetic algorithm and a brute-force method, for model choice. As a demonstration, the C1C2 was applied to simulated and real-world datasets. A penalized linear model was assumed to reasonably approximate the true relation between the dependent and independent variables, thus reducing the model choice problem to a matter of variable selection and choice of penalizing parameter. We also studied the impact of assuming prior knowledge about the number of relevant variables on model choice and generalization error estimates. The results obtained with the C1C2 were compared to those obtained by employing repeated K-fold cross-validation for choosing and assessing a model. Results The C1C2 framework performed well at finding the true model in terms of choosing the correct variable subset and producing reasonable choices for the penalizing parameter, even in situations when the independent variables were highly correlated and when the number of observations was less than the number of variables. The C1C2 framework was also found to give accurate estimates of the generalization error. Prior information about the number of important independent variables improved the variable subset choice but reduced the accuracy of generalization error estimates. Using the genetic algorithm worsened the model choice but not the generalization error estimates, compared to using the brute-force method. The results obtained with repeated K-fold cross-validation were similar to those produced by the C1C2 in terms of model choice, however a lower accuracy of the generalization error estimates was observed. Conclusion The C1C2 framework was demonstrated to work well for finding the true model within a penalized linear model class and accurately assess its generalization error, even for datasets with many highly correlated independent variables, a low observation-to-variable ratio, and model assumption deviations. A complete separation of the model choice and the model assessment in terms of data used for each task improves the estimates of the generalization error.


Background
A common task in computational biology/bioinformatics and computational chemistry/chemometrics (hereafter abbreviated BBCC) is to model a dependent variable from a set of independent variables; this gives insight into the workings of the process being modeled and enables prediction of future observations. Typical examples include analyzing potential drug activity through proteochemometrics and quantitative structure-activity relationship (QSAR) modeling [1][2][3], discovering gene regulatory binding-site modules [4], and predicting clinical outcomes of cancer from gene expression data [5]. However, recent articles have indicated that predictive modeling approaches have not fully fulfilled expectations for solving real problems. This issue has for instance been discussed in the fields of QSAR [6] and microarray gene expression data modeling [7,8]. While some of the problems may be attributed to incorrect use and interpretation of the models, others can be ascribed to improper model selection and assessment. Our aim is here to address the latter issue by introducing the C 1 C 2 , a general framework for model choice and assessment.
Let D = {X n , y n } be a dataset, where X n = (x' 1 , ..., x' n )' is an n × p X matrix whose ith row, x i , is the value of a p X -vector of independent variables associated with y i , the ith row of the n × 1 matrix, y = (y 1 , ..., y n )'. A statistical model, can be used to characterize the relation between X n and y n .
In general, given the dataset D, must be chosen from a set of models, according to some criterion (typically the minimization of a loss function). The most common way to select from in BBCC is to use K-fold cross-validation; that is, the dataset D is split into K mutually exclusive subsets, D 1  where M is the number of models in . This model selection method may seem straightforward and intuitive, however it neglects the fact that all the data at hand is used to make the model choice. Thus, we no longer have an independent testset to assess the chosen model by. The result is that, typically, C 0 ( ) underestimates the generalization error (see for instance [9]), defined as the expected prediction error over an independent test sample. This problem has been highlighted in relatively recent works [9,10], but was noted initially in 1974 [11]. To obtain a more accurate generalization error estimate, the model selection process must be separated from the model assessment in terms of the data that is used. Ideally, if data were abundant and easily produced, we would set aside a large test dataset and use it to assess -but not to choose! -the model , and subsequent model refinements could be assessed with new, unseen data. In practice, this is however often impossible since BBCC data is typically scarce, and expensive to produce. The luxury of large independent testsets can thus rarely be afforded. To tackle this problem, Freyhult et al. [9] suggested using a Kfold cross-validatory assessment of an H-fold cross-validatory model choice, , as a way of simultaneously choosing and assessing its performance; thereby separating the model selection from its assessment. The model is assessed by the function where (D -k ) is the cross-validatory choice of based on D -k ; that is, the model that minimizes the function where D -hk denotes the dataset D with the kth and hth subsets omitted. In the present work we build on and generalize this idea into the C 1 C 2 framework. In general, the number of models in is huge, thus it is unfeasible to go through even a small subset of them manually. Hence, for a framework such as the C 1 C 2 to be useful in practice, automated methods for searching the model space are necessary; in this sense the C 1 C 2 is similar to the automatic tion and assessment. In the interest of clarity, we have restricted our attention to the study of model choice and assessment within a linear model class, (defined below) for a quadratic loss function. We discuss the results of the demonstrations, the pros and cons of the generality of the C 1 C 2 , and set out some directions for further research.

Algorithm
Let C 1 , C 2 ∈ C = {C 1 ,..., C J }, where C is a set of model assessment criteria and C 1 , C 2 represent two specific criteria (i.e. C 1 = C i , C 2 = C j , i, j = 1,..., J). Further, let S ∈ S, where S is a set of search methods; let L ∈ L, where L is a set of loss functions; let G denote a sequence of data processing steps (e.g. mean-centering, transformations, whitening, etc) and let G' contain the results of G applied to D -k (the roles of G and G' are exemplified in the discussion following the pseudococe below); let R be a positive integer and K an integer between 1 and n, where n is the number of observations. The C 1 C 2 procedure is outlined with the following pseudocode:

Initiate
, G, L, C 1 , C 2 , R, and K. c. Search using the data D -k and C 2 as objective function. Assume is found to maximize (or mini- e. Assess using C 1 and D k . Save assessment result. } } The data partitioning in (a) separates data for the model choice from data for the model assessment. Note that the partitioning is dependent on the choice of C 1 and does not necessarily need to be done in a cross-validation fashion. For instance, the choice C 1 = ".632 estimator" [15,16], partitions the data by independently sampling n rows from D with replacements and lets the observations not included among the sampled observations constitute the test set. The output from the C 1 C 2 is also dependent on the choice of C 1 ; for example, the choice C 1 = C 2 = Bayesian Information Criterion (BIC, see [17] and Methods) would not give a direct estimation of the generalization error, but rather an assessment of model overfitting. To clarify the roles of G and G', we give the following example: Let G only contain a processing step that scales to unit variance. In (b) G is applied to D -i and the standard deviation of each column of D -k is saved in G'. In (d), G' is applied to D k , that is, the columns in D k are scaled using the standard deviations calculated in (b). This treatment of G ensures that D k indeed constitutes an independent testset. The 'for loop' over r is introduced to enable calculation of confidence intervals for estimates and, by averaging the estimates over R repetitions, it permits reduction of the variance in parameter and error (or overfitting) estimates by a factor of 1/R. Figure 1 gives a graphical gives a graphical view of the C 1 C 2 framework. We emphasize that the generality of the framework allows C 1 , C 2 , and S to be chosen to fit the problem at hand. Adequate choices of C 1 , C 2 , and S make the model selection and assessment more accurate and faster, which we will discuss below.

Datasets used in the testing
Both the simulated and the real data used for evaluating a new method or algorithm should reflect typical dataset properties found in real-world application domains. Examples of such properties in BBCC are multicollinearity, a large number of independent variables relative to the number of observations, and binary and categorical independent variables.

Simulated data
We simulated datasets as follows: iff is in the model else Three δ i in Δ were chosen to be nonzero and equal to one; the positions were chosen at random to be 11, 14, and 18 (but remained fixed throughout the experiment for evaluation purposes). The corresponding regression coefficients, β 11 (δ 11 ), β 14 (δ 14 ), and β 18 (δ 18 ), were, for simplicity, all set equal to 1. The variables 11, 14, and 18 were slightly correlated with an estimated covariance matrix .
Datasets were generated assuming that y n followed a linear model according to: y n = X n β p (Δ) + ε n , where ε i~N (0,1.5).
Four datasets were simulated in order to evaluate the C 1 C 2 s performance in settings where n <p x , n > p x , and where the observations were sampled from an orthogonal multivariate normal distribution or not, according to the following schema: The simulated datasets are available in CSV format from Additional files 1, 2, 3, 4.

The Selwood dataset
This is a real dataset, made available from a website [18] and originally published in 1990 [19]. It is a widely studied dataset (see [20,21] and references therein). It contains one dependent variable, 53 independent variables, and 31 observations. The 53 independent variables correspond to numerical descriptions of molecules (antifilarial antimycin analogues) designed to capture their physicochemical properties. The dependent variable is the in vitro antifilarial activity of the molecules. This dataset exhibits extremely strong correlations between the independent variables and contains real valued, binary and categorical independent variables. It is known from previous studies that this dataset contains nonlinearities, but that decent models can be found using linear methods.

Testing
To demonstrate the use of the C 1 C 2 , it was applied to the simulated and real-world datasets described above (hereafter referred to as "the datasets"). Below we describe the choices for R, K, , G, L, C 1 , C 2 , and S and the motivation for each selection. .

M
The C 1 C 2 framework Figure 1 The C 1 C 2 framework. The data partitioning in step (a) in the pseudocode separates the model choice from its assessment, which is highlighted in purple in the figure. The left side of the figure relates to steps (b) to (d) in the pseudocode, and the right side to step (e); i.e. the left side relates to choosing the model and saving the parameter estimates, and the right side to assessing the model and saving the assessment results.
constrained by time and is dependent on the size of the dataset and the computational complexity of the choices of , G, L, C 1 , C 2 , and S. The choice of K is a trade-off between bias and variance; the larger the K, the more variance and the less bias in the estimates of the generalization error [22]. R was here set to 12 and K to 5.

Choice of
We make the assumption that a normal linear model forms a reasonable approximation of the data. This model is given by: y n = X n β p + ε n , where the subscripts denote the number of rows in a matrix, β p are regression coefficients, and ε n~N (0,σ 2 I). Further, because n <p and the columns in X n are highly correlated in some of the datasets, we decide to use the ridge estimator, (see Methods), of the regression coefficients, β p . Let be the linear class of models given by: y n = X n β p + ε n , where β p is estimated by . We thus choose . The problem of model choice within reduces to the problem of variable selection, i.e. choosing which subset of the p columns in X n to include in the model, and the problem of choosing the ridge penalizing parameter λ (see Methods). Hence, letting Δ = (δ i ) i = 1,..., p (see simulated data above) represent a subset of X n , we want to choose Δ and λ using the C 1 C 2 framework. A choice of Δ and λ for given values of r and k will be termed "an estimate" of Δ and λ, respectively, and be denoted and . Averages of estimates over the K folds and the R repeats in the C 1 C 2 are denoted and , respectively.
Choice of G As the columns in the Selwood dataset are measured in different units using different scales, we choose to make G contain mean centering and scaling to unit variance processing steps.

Choice of L
We use the standard quadratic loss function given by: Choice of C 1 and C 2 Others [12,13] have suggested choosing C 1 = C 2 = crossvalidation. Here, we choose C 1 = cross-validation and C 2 = BIC. Hence, in this demonstration we assess a model choice according to: where (D -k ) is the chosen according to BIC based on D -k ; that is, the value of that optimizes the function: (5) is the number of free parameters in the model (note that this is not equal to the number of parameters in the model , see for instance [16]).
The choice of C 1 is motivated by that we wish to get a direct estimate, , of the generalization error, ε gen , of our model choice. Provided that the assumptions behind BIC are fulfilled, the choice C 2 = BIC has several advantages over C 2 = cross-validation, including: a reduction of bias in parameter estimates [22], a reduction of variance by the Rao-Blackwell type relation derived in [23], and a drastic reduction of the computational cost of the procedure.
Choice of S A genetic algorithm (GA) was chosen as a search method because it is very easy to adapt to different situations and in general effective for nondeterministic polynomial-time hard combinatorial problems, such as the problem of estimating Δ [24]. A trial solution in the GA is here a varying length chromosome that contains a real-valued number representing λ and a vector of integers representing the indices of the δ i in Δ that are nonzero. The fitness function is our choice of C 2 . For the simulated datasets, we also chose to run the C 1 C 2 with a brute force search method: for each λ ∈ {0,0.01,0.02,...,10} an exhaustive search in the Δ space (i.e. an all-subset regression) was performed. This enabled comparisons between the GA method and a search method guaranteed to find the optimal model (given a specific objective function and the resolution and limits of λ).

Some remarks regarding the demonstration
To enable comparisons with the estimates of Δ, λ, and ε gen obtained with repeated K-fold cross-validation, the demonstration described above was repeated with the func- tion (1) used as a criterion for model choice and for assessing the model. Note that, since the C 1 C 2 includes the 'for loop' over r, (1) was repeated R = 12 times, each time with a new, independent data partitioning. This was done to facilitate an impartial comparison between the two methods.
The demonstration of the C 1 C 2 framework can be compared with the work of for instance Nicolotti and Carotti [20], where a genetic algorithm was employed to estimate Δ. In contrast to that approach, the C 1 C 2 framework completely separates model choice and assessment whereby more accurate generalization error estimates in general are achieved. Further, the use of specific ad hoc objective functions is avoided by choosing C 2 to be a formally derived model selection criterion, and simultaneous estimation of model parameters other than Δ (for example, the ridge parameter λ in the demonstration) can be afforded. Typically, in works that have employed a search method for estimating Δ, a given number of nonzero δ i in Δ is assumed (see for instance [20,25]). Therefore it was of interest to investigate the effect of this assumption on producing good estimates of Δ and ε gen . This can be tested for the simulated datasets in the demonstration, where the number of nonzero δ i is known. The C 1 C 2 was therefore applied to the simulated datasets both with an assumption about the number of nonzero δ i and without the assumption. For simplicity (however somewhat unrealistically), we assumed the correct number of nonzero δ i .

Simulated datasets
The four simulated datasets in combination with the use of either the C 1 C 2 or repeated K-fold cross-validation for model choice and assessment, the GA or the brute-force search method, and either with or without the assumption of prior knowledge of the number of nonzero δ i constitute a two-level, five-factor, full factorial experimental design. The C 1 C 2 and the repeated K-fold cross-validation were applied four times to each factor combination, thus providing four replicates of the whole demonstration for the simulated data. The design can be analyzed within the normal linear model  − % rem), homoscedasticity, and independent errors (data not shown). A few outliers were however observed, probably resulting from "unfortunate" data partitions.

Selwood dataset
Applying the C 1 C 2 to the Selwood dataset yielded that on average contained 11.4 ± 3.5 nonzero . The 11 most frequently picked variables were the partial atomic charge for atoms ATCH1, ATCH3, and ATCH6; electrophilic superdelocalizability for atom ESDL3; the van der Waal's volume VDWVOL; the surface area SURF_A; the principal moments of inertia MOFI_Y and MOFI_Z; the principal ellipsoid axis PEAX_X; the partition coefficient LOGP; and the sum of F substituent constant SUM_F (see [19] for more details about the variables). The estimation of λ by was 2.50 ± 0.09 and the estimation of the generalization error by was 0.42 ± 0.038, where is an average over the R·K produced in the C 1 C 2 .

Implementation
Computer programs to implement the C 1 C 2 were written in Java (Sun Microsystems [26]) as a part of the library P, that will serve as the data analysis plugin for Bioclipse [27]. P is available under the GSPL license from the website [28]; it is open source and free for academics. It has a modular architecture that enables plugging in new features, including modeling methods, model selection criteria, and search procedures. P relies on a modified version of the JGap library (available from the website [29]) for the genetic algorithm computations (the modifications are available under the LGPL license from the website [30]). The R-package, pvclust [31,32], was used for the cluster analysis (see Discussion).

Simulated datasets
The model (6) fitted to w = (see Table 1  c1c2 -the C 1 C 2 was used (as opposed to repeated K-fold crossvalidation), ga -the GA search method was used (as opposed to the brute force search method), cor -correlated independent variables in the dataset (as opposed to uncorrelated,   should provide worse estimates of Δ, whereas more observations and trustworthy prior knowledge should provide better estimates. A significant improvement in average Δ estimates was observed when using the brute-force search compared to the GA. The GA on average selected slightly more variables than needed and than what the brute-force method did. No clear significant difference could be seen between using the C 1 C 2 rather than repeated K-fold crossvalidation. It can be shown that the optimal choice of λ (in terms of minimized expected generalization error) tends to zero as the number of observations tends to infinity and decreases with decreasing number of variables (see for instance [33]) and with decreasing correlations between the independent variables. The model (6) fitted with w = as a response (see Table 2) showed that the average estimated λ was significantly smaller for the data that is not known, these results are thus consistent with theory and provide evidence that both the C 1 C 2 and repeated Kfold cross-validation gave reasonable estimates of λ in the demonstration. However, the average λ estimates are not equal to zero for all orthogonal datasets, presumably due to the stochastic nature of the GA and to errors in y n . No significant differences were observed between using the GA or the brute force search methods or between the C 1 C 2 and the repeated K-fold cross-validation.
Fitting model (6) with w = as a response (see Table 3) showed that the average error estimates were significantly worsened with the assumption of a given number of nonzero δ i and that no significant difference was observed when using the GA or the brute force method, or when the independent variables in the dataset were correlated or not. These findings might seem confusing given that the assumption of a given number of nonzero δ i , the use of the brute-force search method, and uncorrelated independent variables all improved model selection. The findings can be explained by the fact that, in general, without an assumption of a given number of nonzero δ i , when using the GA for searching the model space, and when independent variables were correlated, more nonzero δ i are on average selected (see Table 1).
Thus the chances of also selecting the correct ones improve. This implies that it is worse to estimate at least one δ i = 1 with = 0 than to estimate all δ i = 1 with = 1 and at least one δ i = 0 with = 1. This makes sense, because the former models are incorrect, whereas the latter ones contain the true model, but are inefficient due to their unnecessary large size. The average error estimates were significantly improved with a large number of observations and when the C 1 C 2 was employed to produce the estimates compared to when the repeated K-fold cross-validation was used (see Fig. 2 and Table 3). The latter result seems contradictory with that no clear difference was found between the average Δ estimates produced with the C 1 C 2 and those obtained with the repeated K-fold crossvalidation (see above). It can however be explained by studying the R·K individual Δ estimates, where a clear (99% level) positive effect could be observed when using the C 1 C 2 compared to the repeated K-fold cross-validation. The individual Δ estimates were thus worse when repeated K-fold cross-validation was used, resulting in worse generalization error estimates. However the average Δ estimates from the respective method were almost the same. This observation is seconded by the higher confidence intervals of the average Δ estimates produced with repeated K-fold cross-validation (see Additional file 5). The finding that the C 1 C 2 produces more accurate generalization error estimates than repeated K-fold cross-validation is consistent with the results presented in for instance [9] and provides evidence for that a complete separation of the data used for model choice and the data used for model assessment is necessary to obtain better estimates of the generalization error.

Selwood dataset
The result of estimating Δ was, expectedly, less clear when applying the C 1 C 2 and repeated K-fold cross-validation to real-world data; the Selwood dataset is particularly difficult to model due to the extremely high correlations between variables (many variable pairs have correlation coefficients > 0.95), the low observation-to-variable ratio, and deviations from the linearity and homoscedasticity assumptions (see [20]). 11.4 out of the 53 variables were on average selected by the C 1 C 2 and 14.1 by K-fold crossvalidation. Interestingly, the 11 most frequently picked variables selected by the C 1 C 2 is a proper subset of the 14 most recurrently selected variables by K-fold cross-validation. Hierarchically clustering the 14 most frequently picked variables chosen by K-fold cross-validation (which thus includes the 11 variables selected most often by thê lê e gen gen − %d id î d i C 1 C 2 ) using the absolute correlation as a distance measure revealed three distinct clusters and one subcluster (see Fig.  3). Good models (in terms of estimated generalization error) for the Selwood dataset can be achieved by selecting LOGP and one variable from the set of variables in the blue subcluster (PEAX_X, MOFI_Y, MOFI_Z, VDWVOL, and SURF_A) and one from the set of variables in the green cluster (ESDL3, ATCH1, ATCH3, ATCH6, ATCH7, and SUM_F). LOGP appears to be sufficiently different from the rest of the variables in the red cluster to improve model performance. The variables in the respective blue and green clusters are highly correlated and it is sufficient to have one variable from each cluster in a model.
Models containing LOGP and one variable each from the green and blue clusters have high predictive power and comply to the QUICK rules for credible predictive models proposed previously [21]. Furthermore, these models have been found credible in the works of others [20,25,34]. The C 1 C 2 chose 11 variables belonging to the green, red, and blue clusters, whereas K-fold cross-validation chose an additional three variables: ATCH7 in the green cluster, and DIPMOM and DIPV_Y belonging to the third distinct (yellow) cluster. Variables from the yellow cluster do not improve the internal predictive ability when testing models containing LOGP and one variable from the respective green, blue, and yellow clusters on the whole Selwood dataset (data not shown); this result is supported by the work of Nicolotti and Carotti (see Table  1 in [20]). More variables in the Selwood dataset were thus on average selected with repeated K-fold cross-validation than when using the C 1 C 2 (the difference was significant on the 80% level, tested by a one-sided Welch's ttest), including two that not seem to improve the predictive ability of the models. The generalization estimates obtained with the K-fold cross-validation was lower than those obtained with the C 1 C 2 (significant on the 70% level). Although these differences are not highly significant, it is tantalizing to arrive at the conclusion that the models selected by repeated K-fold cross-validation in this Generalization errors obtained with the C 1 C 2 and repeated K-fold cross-validation Cluster dendrogram of the 14 selected variables from the Selwood dataset using repeated K-fold cross-validation. Three distinct clusters can be noted (shown in red, green, and yellow rectangles). One sub-cluster can be seen within the red cluster (shown in a blue rectangle). The red and green numbers are p-values of a given cluster; they indicate how well the cluster is supported by data (see [31] for details). + Additional variables selected by repeated K-fold cross-validation compared to the C 1 C 2 .
particular case are more prone to overfitting and that this is reflected in the lower generalization error estimates.
The C 1 C 2 We have here introduced the C 1 C 2 framework for simultaneous model choice and assessment. The main idea is a complete separation of the choice of a model and its assessment in terms of the data used for each task. The C 1 C 2 was applied to the problem of choosing a model, . Previously, others have described methods that, within the linear model, tackle the problem of regression coefficient shrinkage and variable selection simultaneously, for example, the lasso [35]. However, the C 1 C 2 framework is general and is easily applied to other settings. For instance, different choices of C 2 are favorable in different situations; Akaike's information criterion (AIC) [36] is known to be consistent within the linear model if the true model is very complex, whereas BIC is favorable within linear models of finite dimension [37], and cross-validation is preferable to use in situations where the degrees of freedom of a model is difficult to define, and so forth. The search method can also be tailored to the problem at hand; for instance, brute-force methods are advantageous for small problems, whereas GAs are faster and thus applicable to larger problems. Moreover, if required, can be chosen to contain nonlinear models, L can be chosen to be exponential in order to increase the penalty on outliers, and instead of using the search method to produce an estimate of Δ (as we did in the demonstration) we can let G contain a dedicated variable selection method. The cost of this generality is uncharacterized convergence rates (in finite time) of the parameter estimates, which is coupled to the need of employing a general search method (like a GA) rather than solving standard convex problems. Running the C 1 C 2 R times enables averaging of estimates and calculation of confidence intervals, but renders problems in choosing which out of the R models to use for interpreta-tion and future predictions. A potential remedy to these problems is, instead of choosing a model, to employ all chosen models in a stacking-like schema (see [38] for details on stacking). Testing this idea and further testing of the C 1 C 2 for other choices of , G, L, C 1 , C 2 , and S will be pursued in future research.

Conclusion
We have presented some evidence that suggests that the C 1 C 2 works well in terms of choosing the correct model and produce good estimates of the generalization error. It was demonstrated to perform well within a penalized linear model, even for "difficult" datasets with highly correlated independent variables, a low observation-tovariable ratio, and deviations from model assumptions (see Table 4 for a summary of the findings in the demonstrations). However, more research is needed to fully assess the methods performance for more general, for instance nonlinear, models and to provide theoretical insight to frameworks such as the C 1 C 2 . The C 1 C 2 is general and reasonable choices of , G, L, C 1 , C 2 , and S help in achieving as unbiased estimates with as low a variance to as low a computational cost as possible. A framework that completely separates model choice from assessment in terms of used data, like the C 1 C 2 , should always be employed for model selection and assessment in order to avoid positive bias in the generalization error estimates and, ultimately, to avoid false conclusions and using dubious models to direct further research.

Bayesian Information Criterion (BIC)
Suppose we have a set of candidate models, and cor-  Both the C 1 C 2 and repeated K-fold cross-validation performed well at finding the true Δ (even when independent variables are highly correlated and when n <p).
The C 1 C 2 and repeated K-fold cross-validation produced reasonable estimates of λ.
Prior information about the number of important independent variables improves model choice but can reduce the accuracy of generalization error estimates. Correlated independent variables and using the genetic algorithm worsened the model choice significantly, but not the generalization error estimates.
The C 1 C 2 compares favourably with repeated K-fold cross-validation for assessing the generalization error.
n denotes the number of observations in a dataset, p the number of variables, Δ represents a given subset of the p variables, and λ the ridge regression parameter.
To choose a model in a Bayesian setting, the posterior odds, given by: are formed for all models , and is picked to maximize equation (7). If all models in are given equal prior probabilities, the problem of choosing the model is reduced to calculating the integrals, P(D| ). It can be shown [39,40] that the Bayesian Information Criterion (BIC) approximates the logarithm of this integral with an O(1) error term, that is: where In the latter expression, is the maximum likelihood estimate, df m is the number of free parameters in model (note that this in general is not equal to the number of parameters in the model), and n is the number of observations [17]. Thus, BIC is an approximation to the Bayes solution, but valid outside the Bayesian context. This is true because the leading terms in the approximation do not depend on the prior densities of the model parameters, θ m . BIC is, as opposed to nonparametric approaches such as cross-validation, model based and therefore relies on the assumptions made in the modeling. BIC is derived under the assumption that the data comes from a distribution in the exponential family (see [41] for more about the assumptions behind BIC and a comparison with Akaike's Information Criterion).

Ridge regression
The ordinary least squares (OLS) estimator of the regression coefficients β in the standard linear model is efficient (i.e. has the minimum possible variance) within the class of linear and unbiased estimators. However, when the independent variables are correlated, the variance of the OLS estimator is generally high. In these situations, ridge regression [42] can yield improved parameter estimates by minimizing a penalized residual sum of squares, given by: RSS (λ) = (y -Xβ) T (y -Xβ) + λβ T β. Finding the minimum of this expression gives the ridge solution: = (X T X + λI) -1 X T y, where I is the p × p identity matrix. The solution thus adds a positive constant to the diagonal of X T X before inversion; this makes the problem nonsingular, even if X T X is not of full rank. While this introduces bias into the coefficient estimates, variance is often greatly reduced.
Note that is a linear function in y, thus it is straightforward to define the effective degrees of freedom of the ridge regression fit, (df(λ)) as: df (λ) = tr [X(X T X + λI) -1 X T ] [16]. The degrees of freedom of the fit are needed for carrying out model selection according to, for instance, BIC. Linearity in y also enables easy implementation (no quadratic programming required as, for instance, is necessary with the lasso).

Genetic algorithm (GA)
A GA (see [43] for more details) is a stochastic search technique for finding exact or approximate solutions to optimization and search problems. A typical genetic algorithm is defined by a genetic representation of a given solution (normally termed a chromosome in the GA context). That is, a vector, specifies the numerical representation of the ith chromosome at generation t, and an objective function (or fitness function), f( )→/ &#x211D;/evaluates the fitness of a chromosome. The GA is initiated by setting up a random population that contains a number of trial chromosomes. New solutions are generated by mutation or recombination of existing solutions and are selected for the next generation with a probability given by: . The process is continued through a number of generations until an optimal or acceptable solution has been found. Genetic algorithms of this type can be shown to converge with a probability of one to the global optimal solution as t → ∞.

Brute force search
A brute force search systematically tests an exhaustive list of all possible candidates for the solution to a given search or optimization problem and checks whether each candidate satisfies the problem's statement.