Research article  Open  Published:
Unbiased descriptor and parameter selection confirms the potential of proteochemometric modelling
BMC Bioinformaticsvolume 6, Article number: 50 (2005)
Abstract
Background
Proteochemometrics is a new methodology that allows prediction of protein function directly from real interaction measurement data without the need of 3D structure information. Several reported proteochemometric models of ligandreceptor interactions have already yielded significant insights into various forms of biomolecular interactions. The proteochemometric models are multivariate regression models that predict binding affinity for a particular combination of features of the ligand and protein. Although proteochemometric models have already offered interesting results in various studies, no detailed statistical evaluation of their average predictive power has been performed. In particular, variable subset selection performed to date has always relied on using all available examples, a situation also encountered in microarray gene expression data analysis.
Results
A methodology for an unbiased evaluation of the predictive power of proteochemometric models was implemented and results from applying it to two of the largest proteochemometric data sets yet reported are presented. A double crossvalidation loop procedure is used to estimate the expected performance of a given design method. The unbiased performance estimates (P^{2}) obtained for the data sets that we consider confirm that properly designed single proteochemometric models have useful predictive power, but that a standard design based on cross validation may yield models with quite limited performance. The results also show that different commercial software packages employed for the design of proteochemometric models may yield very different and therefore misleading performance estimates. In addition, the differences in the models obtained in the double CV loop indicate that detailed chemical interpretation of a single proteochemometric model is uncertain when data sets are small.
Conclusion
The double CV loop employed offer unbiased performance estimates about a given proteochemometric modelling procedure, making it possible to identify cases where the proteochemometric design does not result in useful predictive models. Chemical interpretations of single proteochemometric models are uncertain and should instead be based on all the models selected in the double CV loop employed here.
Background
Current computational methods for prediction of protein function rely to a large extent on predictions based on the amino acid sequence similarity with proteins having known functions. The accuracy of such predictions depends on how much information about function is embedded in the sequence similarity and on how well the computational methods are able to extract that information. Other computational methods for prediction of protein function include structural similarity comparisons and molecular dynamics simulations (e.g. molecular docking). Although these latter methods are powerful and may in general offer important 3D mechanistic explanations of interaction and function, they require access to protein 3D structure. Computational determination of a 3D structure is well known to be resource demanding, error prone, and generally requires prior knowledge, such as the 3D structure of a homologous protein. This bottleneck makes it important to develop new methods for prediction of protein function when a 3D model is not available.
Recently a new bioinformatic approach to prediction of protein function called proteochemometrics was introduced that has several useful features [1–4]. In proteochemometrics the physicochemical properties of the interacting molecules are used to characterize protein interaction and classify the proteins into different categories using multivariate statistical techniques. One major strength of proteochemometrics is that the results are obtained directly from real interaction measurement data and do not require access to any 3D protein structure model to provide quite specific information about interaction.
Proteochemometrics has its roots in chemometrics, the subfield of chemistry associated with statistical planning, modelling and analysis of chemical experiments [5]. In particular it is closely related to quantitativestructure activity relationship (QSAR) modelling, a branch of chemometrics used in computer based drug discovery. Modern computer based drug discovery is based on modelling interactions between small drug candidates (ligands) and proteins. The standard approach is to predict the affinity of a ligand by means of numerical calculations from first principles using molecular dynamics or quantum mechanics. QSAR modelling is an alternative approach where experimental observations are used to design a multivariate regression model.
With x_{ i }denoting descriptor i among D different descriptors and y denoting the biological activity, (linear) QSAR modelling aims at a linear multivariate model
y = w^{T}x = w_{0} + w_{1}x_{1} + w_{2}x_{2} + ... + w_{ D }x_{ D } (1)
where w = [w_{0}, w_{1}, w_{2},..., w_{ D }]^{T}are the regression coefficients and x = [1, x_{1}, x_{2},..., x_{ D }]^{T}. The activity y, may be the binding affinity to a receptor but may also be any biological activity e.g., the growth inhibition of cancer cells. In comparison with numerical calculations from first principles and similar approaches, the main advantages of QSAR modelling are that it does not require access to the molecular details of the biological subsystem of interest and that information can be obtained directly from relatively cheap measurements.
The joint perturbation of both the ligand and protein in proteochemometrics yields additional information about the different combinations of ligand and protein properties for an interaction than can be obtained in conventional QSAR modelling where only the ligand is perturbed. In recent years, various other bioinformatic modifications of conventional QSAR modelling have been reported. These include simultaneous modifications of the ligand and the chemical environment (buffer composition and/or temperature) in which the interaction take place [6–8], and threedimensional QSAR modelling of proteinprotein interactions that directly yields valuable stereochemical information [9].
Although proteochemometrics has already proven to be an useful methodology for improved understanding of biochemical interactions directly from measurement data, the quantitative proteochemometric models designed so far have not yet been subject to a detailed and unbiased statistical evaluation.
A key issue in this evaluation is the problem of overfitting. Since the number of ligand and protein properties available is usually very large, to avoid overfitting, one has to constrain the fitting of the regression coefficients. For example, in ridge regression [10], a penalty parameter is tuned based on data to avoid overfitting, and in partial least squares (PLS) regression [11–13] the overfitting is controlled by tuning the number of latent variables employed. In proteochemometrics as well as in many QSAR studies reported, the performance estimates reported are obtained as follows: 1) Perform a Kfold CV for different regression parameters, 2) Select the parameter value that yields the largest estimated performance value, and 3) Report the most promising model found and the associated performance estimate. Although this procedure may seem intuitive and may yield predictive models (as we in fact demonstrate below) the performance estimates obtained in this way may be heavily biased. Interestingly, this problem was recently addressed in the context of conventional QSAR modelling [14], and has also been discussed in earlier work, see [15, 16].
As an alternative or complement to constraining the regression coefficients, one may also reduce the variance by means of variable subset selection (VSS). In QSAR modelling, many algorithms for VSS have been proposed based on various methodologies, for example optimal experimental design [17, 18], sequential refinements [19], and global optimization [20]. VSS is used to exclude variables that are not important for the response variable, in the process of model building. Variables that are not important receive low weights in both a PLS and a ridge regression model, however if the fraction of unimportant variables is very large [21] the overall predictive power of the model is reduced. In this case VSS can improve the predictivity. However, if the fraction of unimportant variables is rather small, the quality of the model will not be improved by using VSS, it might on the contrary be slightly reduced. However, the interpretability of the model will in both cases be improved.
Although many of the advanced algorithms for VSS are powerful, they are all computationally demanding. Therefore, in order to keep the computing time down in our use of the double loop cross validation procedure employed here, conceptually and computationally simple algorithms for VSS were used instead of the more advanced ones presented, e.g. in [17–20]. Most likely, the more advanced algorithms would yield more reliable models with even higher predictive power than for the models designed here. However, the main issue of interest in this paper is to confirm the potential of proteochemometrics.
In previous reported proteochemometrics modelling, all available examples were used in the VSS. These were split into K separate parts and a conventional Kfold cross validation (CV) was performed. However, since all available examples were used, there were no longer any completely independent test examples available for model evaluation. Interestingly, this problem of introducing an optimistic selection bias via VSS was recently also pointed out in the supervised classification of gene expression microarray data [22].
In this paper we employ a procedure that can be used to perform unbiased statistical evaluations of proteochemometric and other QSAR modelling approaches. An overview of this socalled double loop CV procedure is presented in Figure 1, and may be regarded as a refinement of the current practice in proteochemometrics in the following respects:

1.
K_{1} different variable subset selections are performed, one for each step in the outer CV loop. This avoids optimistic selection bias.

2.
The best performance estimates (Q^{2}) found in the inner loop by means of K_{2}fold CV are computed, but not reported as the model's performance estimate. This avoids the second optimistic selection bias mentioned above.

3.
An unbiased performance estimate, P^{2}, is computed in the outer loop and is reported as the performance estimate of the modelling approach defined by the procedure in the inner loop (the methods of VSS, regression, and model selection employed). P^{2} is the result of different models that are designed and selected in the inner loop. It reflects the performance that one should expect on average.

4.
Repeated K_{1}fold CVs which yield information about the robustness in the results obtained (presented as confidence intervals).
In addition to these refinements, this work also demonstrates the potential of fast and straight forward alternative methods for VSS and regression in the inner loop. Moreover, it indicates that the performance estimates reported by certain software packages for QSAR may be quite misleading.
We reanalyzed two of the largest proteochemometric data sets yet reported. The first data set is presented in [2] and contains information about the interactions between 332 combinations of 23 different compounds with 21 different human and rat amine Gprotein coupled receptors. In total, there are 23 × 21 = 483 possible interactions and the basic task is to fill in the 483332 = 151 missing values. The second data is presented in [23] and contains information about the binding of 12 different compounds (4piperidyl oxazole antagonists) to 18 human α_{1}adrenoreceptor variants (wildtypes, chimeric, and point mutated). As for the first data set, there are not interaction data available for all the 12 × 18 = 216 possible interactions, but for 131, see [24] for more details about this data set. Below these two data sets are referred to as the amine data set and the alpha data set, respectively.
Results
Software
Computer programs were written in MATLAB (Mathworks Inc., USA) to integrate the double loop procedure in Figure 1 with robust multivariate linear regression using partial least squares (PLS) regression and ridge regression. These programs also contained two simple and fast methods for variable ranking called corrfilter and PLSfilter. For details, see the Methods section.
Parameters
The joint variable selection and PLS tuning performed in the inner K_{2}fold CV loop was performed with K_{2} = 5. The different values of N_{ D }(the number of molecular descriptors) evaluated were 10, 20, 50, 100, 200, ..., 1000, 1500, 2000, ..., 6000 for the amine dataset and 10, 20, 50, 100, 200, ..., 1000, 1500, 2000 for the alpha data set. The values of N_{ L }considered were either the number of latent variables 1, 2, ..., 8, for both the amine and alpha data set or the degree of RR penalty 0, 0.5, 1.0, ..., 3.0 for the amine data set and 10, 50, 100, 150, 200 for the alpha data set. In the outer K_{1}fold CV loop, the same number of splits (K_{1} = 5) was used as in the inner loop. On the global level, the complete experiments were performed 100 times using different random partitions of the complete data sets.
Unbiased predictive power
In Table 1 a summary of the results from 100 randomly selected partitions of the complete amine data set are presented in the form of the mean values and standard deviations obtained. The number of molecular descriptors and latent variables selected in the inner loop are summarized in Table 2. The average values of the biased Q^{2} obtained in the inner loops look quite promising for the PLSfilter method (Q^{2} = 0.90 for both PLS and RR) and is even higher than the value reported in earlier studies [2]. However, the corresponding unbiased performance estimate P^{2} is much smaller (P^{2} = 0.52 or 0.51 for PLS and RR, respectively). The Q^{2} values for the models obtained after variable selection using corrfilter are significantly lower than when using PLSfilter, but the P^{2} values are almost on the same level for the two variable selection methods when no variable selection at all is used. corrfilter reduces the number of descriptors to about one third of the initial number, but corrfilter still selects more than twice as many descriptors than PLSfilter (see Table 2). Since the main reason for variable selection is improving the interpretation of the model by reducing the number of descriptors, this indicates that one should select PLSfilter instead of corrfilter. In Figure 2, the external (unbiased) predictions used to compute P^{2} for the PLS model using PLSfilter show that there is useful predictive power, but only for examples with midrange pK_{ i }values. The model has poor predictability for both low and high pK_{ i }values, indicating that the standard design procedure used in the inner CV loop does not always yield reliable models. This confirms earlier findings [14], that maximization of the unbiased performance estimate Q^{2} is not always reliable, and also indicates that unreliable designs can be detected by means of the outer CV loop employed in this work.
The estimated performances of the models for the alpha data sets are presented in Table 3. Here both the Q^{2} and P^{2} values are high and the difference between the two measures is smaller than for the amine data set. This indicates a lower level of overfitting. The number of descriptors selected in the variable selection is much lower for the alpha data set (see Table 4) than for the amine data set. Both the high P^{2} values and the display of the external prediction in Figure 3 show that the models have high predictive power. Also, the predictive power is significantly higher after variable selection than without. This is an example when variable selection does not only improve the model interpretability, but also the the model predictivity. The above results indicate, for example, that a combination of PLSfilter, PLS regression and model selection by maximization of Q^{2} produces individual models with predictive power. The relative standard deviation of the predictive power is less than 5% for the two data sets considered. However, the number of variables selected has a relative standard deviation of 455/1933 = 25% and 69/199 = 35%, respectively. Moreover, the standard deviation in the number of latent variables (an implicit constraint on the regression coefficients) is approximately one (1.4 and 0.8) or 15%. In conclusion, the individual models are quite different but essentially all of them yield useful predictions.
Comparisons to other programs
To verify that our computations using MATLAB are comparable to computations by other programs, such as SIMCA, GOLPE and UNSCRAMBLER, models without variable selection were performed with all four approaches. In the comparison we have compared Q^{2} values for models based on all descriptors built with PLS using between one and ten latent variables for the amine data set. All the Q^{2} values were computed using the leave out CV method with five random groups and are presented in Figure 4.
Remarkably, the Q^{2} values obtained with SIMCA 7.0 are much higher than for the other methods. This is due to the fact that SIMCA does not use the standard formula (Eq. 3) to compute Q^{2} (personal communication with Umetrics), for some general information see [25].
Robustness and interpretability
To study the robustness and interpretability of the set of models obtained using the two data sets considered, two different levels of information were computed and presented. The first level of information consists of two histograms displaying, for each ligand block (L1–L6 for the amine data set, and L1–L3 for the alpha data set (for the alpha data set the three ligand blocks correspond to the three positions of modification in the ligand)), and for each transmembrane region (TM1–TM7), how often different kinds of descriptors are selected. The histograms are based on the 500 observations obtained in the 5fold cross validation performed 100 times using different, randomly selected, partitions of the data set. The descriptors are divided into receptor descriptors and ligand descriptors that are further subdivided into original descriptors, cross term descriptors, and absolute valued cross terms. In Figure 5, hit rates for receptor and ligand blocks in the 100 different 5fold cross validations performed are presented, both for the original, the cross term, and the absolute valued cross term descriptors. In Figure 5 A and 5B, the results for the amine data set are presented. Figure 5 C and 5D displays the corresponding results for the alpha data set.
The second level of information displays the average and the standard deviation of the contribution of the different TM regions in the receptors for creation of receptorligand affinity according to the 500 different models designed. The contributions were calculated exactly as described in [2] for a single model, and then the average and standard deviation were calculated. Therefore, the results presented in Figure 5 corresponds to Fig. 3 in [2] where the results for a single proteochemometric model were presented. As before, for each TM region, the contributions to the affinity by different ligands is displayed, this time the variance (uncertainty) information is added. The top of Figure 6 shows the detailed contributions of TM regions to affinity, for each possible combination of ligand and receptor, according to the 500 different proteochemometric models designed using the amine data set, PLS regression and the PLSfilter VSS algorithm. The bottom part displays the corresponding results for the alpha data set when employing ridge regression and the PLSfilter VSS algorithm.
Discussion
In summary, the results reported here confirm earlier reports on the potential of proteochemometrics modelling for prediction of biological activity. It is interesting to note that the VSS did increase the predictivity of the models for the alpha data set, but not for the amine data set. The VSS for the alpha data set did also reduce the number of variables to approximately 4% of the original variables, while for the amine data set 15–38% of the variables remained after VSS. This indicates that for models where many variables receive low weights (as for the alpha data set) the VSS can significantly improve the model, whereas for a data set like the amine data set, with less low weighted variables, the VSS does not improve the model even though it can improve the interpret ability of the model.
The basic goal of proteochemometric modelling is to obtain a single quantitative model that can predict biological activities accurately and which can be easily interpreted biochemically. In this context, it is important to stress that the only role of the outer loop employed in this work is to obtain unbiased estimates of the average performance of the design procedure considered in the inner loop. The additional random splitting of the data sets is used on top of this to gain information about the stability in the performance estimates. Thus, for procedures in the inner loop that yield small variances around a high average of P^{2}, there is statistical support that a single design will yield useful predictions. In order for a single model to be chemically interpretable as well, all the models selected in the inner loop should yield approximately the same number (same set) of variables and the constraints on the regression coefficients (e.g., the number of latent variable in PLS regression) in all models should be approximately equal. With this in mind, the results presented in this work indicate that it is possible to design single proteochemometric models with predictive power based on the two data sets considered but that there is a relatively large variance (from one design set to another) in the variables selected and the constraints put on the regression coefficients. This indicates that although a single proteochemometric model would be useful for predictions, a detailed chemical analysis of such a model would be uncertain. More reliable information should be gained from a careful joint analysis of all the models (and their variables) selected in the inner loops of the different evaluations performed. For example, as briefly discussed in [9], the variables selected with the highest frequency should be of great interest. Thus, systematic and simultaneous biochemical analyses of all the models selected in the inner loops of this kind are required. For illustrative purposes of the complexity and potential of such analyses, here we have presented frequency distributions indicating which variable blocks are selected frequently in the two modelling problems considered.
Moreover, we have also presented estimates of the variability (uncertainty) in estimating the contributions to affinity, between various combinations of ligands and receptors, from different transmembrane regions. In Figure 5 (top), histograms display how often different kinds of descriptors were selected in the 500 models designed for the amine data set. One conclusions is that for corrfilter, the absolute valued cross terms are selected three times as often as ordinary cross terms. Another conclusion is that for PLSfilter, fewer variables are selected and there is no obvious preference for one of the two types of cross terms. For the alpha data set it is obvious from Figure 5 C that only TM2 and TM5 are important to the model. From Figure 6 C and 6D, it is also obvious that the cross terms (and also the absolute valued cross terms) are selected less often than the ordinary descriptors.
Figure 6 A and 6B displays contributions to affinity decomposed separately for each TM region and each drug/receptor combination. One conclusion here is that there is substantial variance in the estimates of the contributions which now is revealed and should dampen the risk of overinterpretations. Another conclusion is that the different regression and variable selection methods employed give similar results. Therefore, only one result each for the amine and the alpha data sets are presented in Figure 6. A third conclusion is that a more clear and more reliable pattern of contributions can be identified in the present study than from the estimated contributions in [2] which were based on a single model only. For example, a pattern of consistently negative average contribution is found from TM3 and the receptors 5HT1B to 5HT1F, but this pattern does not appear in Fig. 3 of [2]. A fourth conclusion is that for the alpha data set, there seem to be no significant contributions to affinity from TM1, TM3, TM4, TM6 and TM7. This result agree with previous results for this data set [2].
Although earlier findings have been confirmed, one should note that there are a number of differences between the present and earlier studies which makes detailed comparisons difficult: 1) In earlier work different variable subset selection methods were employed and in some attempts there were no subset selection at all. 2) The normalization and use of nonlinear cross terms differ between the present and earlier studies of the alpha data set. 3) The limited forms of external predictions attempted earlier e.g., in [2] are not directly comparable with the present results. 4) Different software packages have been employed for model selection and performance estimation.
Conclusion
This work employs a methodology for unbiased statistical evaluation of proteochemometric modelling and confirms that proteochemometric modelling is a new bioinformatic methodology of great potential. The statistical evaluation performed on two of the largest proteochemometric data sets yet reported indicates that detailed chemical analyses of single proteochemometric models may be unreliable and that a systematic analysis of the set of different proteochemometric models produced in the statistical evaluation should yield more reliable information. Finally, although this work has focused on confirming the potential of proteochemometrics, the kind of systematic unbiased performance estimation employed here is of course also relevant for closely related areas of bioinformatics like microarray gene expression analysis and protein classification.
Methods
Data sets
In the amine data set, each of the 23 compounds was described by means of 236 different GRid INdependent Descriptors (GRIND) [26] computed for the lowest energy conformation found and organized into 6 different blocks associated with different kinds of physical interactions. In addition, each receptor was split into seven separate transmembrane regions by means of an alignment procedure and then each amino acid was described by means of five physicochemical descriptors (zscales). In total, 159 transmembrane amino acids were translated into 795 physicochemical descriptors organized into 7 different blocks (regions). In the alpha data set each of the 12 different compounds was described by means of 24 binary descriptors indicating the presence of different functional groups at three positions in the compound. Moreover, 52 amino acids in the transmembrane regions of the receptors were identified to have varying properties between receptors and each of them were also coded into five or two physicochemical properties each, yielding totally 96 descriptor values.
Before the proteochemometric modelling step, the amine data set was subjected to preprocessing in order to reduce the dimensionality of the original descriptors. This step should be part of the design procedure, leaving external examples outside. However, this issue is not expected to be critical and was therefore ignored in this study. For the compounds in the amine data set, after mean centering (no normalization), principal component analysis (PCA) was employed separately to each of six different blocks of GRIND descriptors, each block representing a particular kind of physical interaction. Similarly, each of the seven transmembrane receptor block descriptors was subjected to PCA. This resulted in 6 × 10 = 60 compound descriptors and 7 × 15 = 105 receptor descriptors. Finally, 12,600 additional "crossterm" descriptors were produced by combining the compound and receptor descriptors nonlinearly. The crossterms were added to account for nonlinearities and they are shown to significantly improve the model predictivity. For each pair of compound and receptor descriptor blocks (totally 6 × 7 = 42 pairs), the 150 possible products between a compound and receptor descriptor value were computed. In addition, the absolute value of the deviation of each product from the average of the product over the data set available was computed. This resulted in 300 descriptor values for each of the 42 block pairs i.e., 42 × 300 = 12,600 values. For the alpha data set, the cross terms formed were the 2 × 24 × 96 = 4,608 possible products between the descriptors of ligands and receptors. No blockwise PCA was employed to reduce the dimensionality.
As a final step before entering the modelling phase, all descriptor values were mean centered and normalized to have unit variance.
Robust PLS and ridge regression
In PLS regression, first a latent variable model
x = t_{1}b_{1} + t_{2}b_{2} + ... + t_{ M }b_{ M } (2)
of the vector x of descriptor values is created where t_{ m }is latent variable and b_{ m }is the corresponding basis (loading) vector. As few uncorrelated latent variables as possible which have the largest covariances with the response variable y, are selected. Then, a linear model y = a_{0} + a_{1}t_{1} + ... + a_{ M }t_{ M }is obtained from ordinary least squares fitting. Usually, this predictor is transformed back into the original variables yielding y = w^{T}x as in (1). The robustness of PLS comes from the latent variable modelling which eliminates problems caused by strongly correlated variables and few examples. Ridge regression achieves its robustness by adding a penalty term (or, equivalently, a Bayesian prior) to the ordinary least squares criterion that reduces the variances in the regression coefficients. In the experiments considered below, the degree of penalty used in the RR and the number of latent variables used in the PLS regression were tuned in the inner CV loop to maximize their corresponding inner K_{2}fold cross validation performance estimates.
Variable ranking algorithms
In the PLS modelling, the subsets of molecular descriptors used were selected jointly with the latent variables. Before the joint selection was performed, the molecular descriptors were ranked using two simple and fast methods: A bottomup algorithm, which we call corrfilter, and a topdown algorithm which we call PLSfilter, corrfilter ranks the molecular descriptors according to the Pearson correlation coefficient between the descriptor and the response variable (the affinity). PLSfilter first builds a PLS model using all available descriptors and between one and L latent variables, where L is the number of latent variables associated with the model in (2) that explain 99% of the observed variance in y. Then each descriptor is ranked according to the corresponding mean of the squared coefficients, w_{ i }, in the regression models (1) from the L different models. For the alpha data set below only PLSfilter is applicable. This is due to the discrete nature of the ligand descriptors.
Inner loop: joint VSS and regression parameter selection
After completing the variable ranking, the most promising combination of the number of topranked variables and the number of latent variables in the PLS regression modelling or the degree of penalty in the ridge regression modelling was selected as judged by a K_{2}fold CV performance estimate. The performance estimates for different combinations of values of N_{ D }, the number of topranked molecular descriptors, and values of N_{ L }, the number of latent variables (PLS) or degree of penalty (RR), were considered. Finally, the pair (, ) of numbers yielding the highest estimated predictive power was selected.
The predictive power of the models was measured by the commonly used dimensionless quantity Q^{2} defined as
where n is the number of examples, y_{ i }is the measured biological activity of example i, is the corresponding prediction, and is the arithmetic mean value of all the measured activities. Hence, Q^{2} is a CV estimate of the fraction of the variance of the response variable explained by the model. In the case of ordinary least squares fitting, Q^{2} is also a CV estimate of the squared Pearson correlation coefficient between the true (y) and the predicted () response values. Thus, a value of Q^{2} close to one is traditionally interpreted as a good (valid) model.
Outer loop: external K_{1}fold CV
As already mentioned, selection of a QSAR model that maximizes a K_{2}fold CV performance estimate is common in conventional chemometrics and is also applied in proteochemometrics. This method of tuning is more complicated and therefore slower than simpler alternatives (such as tuning to maximize a single conventional hold out performance estimate) but is expected to be less sensitive to overfitting. Although parameter tuning based on CV is attractive, overfitting may still occur and the performance estimate obtained may be too optimistic. Some aspects of this danger were recently pointed out [14] and has also been discussed in much earlier work [15]. In conclusion, it is important to employ a second external CV as in Figure 1 to estimate the true performance also of sophisticated design procedures that employ CV for parameter tuning.
For each step in the external K_{1}fold CV loop, one of the K_{1} subsets of the whole data set was kept for validation and the rest were used for design of a regression model. The predictions obtained in this outer CV loop were finally used in the formula for Q^{2} in (3). However, since the predictions used for calculating Q^{2} were kept outside the whole design procedure, as in earlier work [9, 16], we denote the computed quantity by P^{2} to indicate that this is an unbiased performance estimate based on external predictions.
Repeated K_{1}fold CVs
The results obtained from a single K_{1}fold CV are interesting but are sometimes heavily influenced by the particular data partitioning used. In the work reported here, we therefore performed repeated K_{1}fold CV in the outer loop. For each partitioning selected randomly, the corresponding value of P^{2} was computed using the procedures described above. Thus, a set of different values of P^{2} were obtained and used for determination of the variability in the results obtained.
Computations
The main body of programming and computations were performed using MATLAB on standard processors (900 MHz). For comparisons, we also employed the program packages SIMCA (Umetrics, Sweden), GOLPE [17] and UNSCRAMBLER (CAMO, Norway).
References
 1.
Prusis P, Lundstedt T, Wikberg JE: Proteochemometrics analysis of MSH peptide binding to melanocortin receptors. Protein Eng 2002, 15: 305–311. 10.1093/protein/15.4.305
 2.
Lapinsh M, Prusis P, Lundstedt T, Wikberg JE: Proteochemometrics modeling of the interaction of amine Gprotein coupled receptors with a diverse set of ligands. Mol Pharmacol 2002, 61: 1465–1475. 10.1124/mol.61.6.1465
 3.
Wikberg JE, Mutulis F, Mutule I, Veiksina S, Lapinsh M, Petrovska R, Prusis P: Melanocortin receptors: ligands and proteochemometrics modeling. Ann N Y Acad Sci 2003, 994: 21–26.
 4.
Wikberg J, Lapinsh M, Prusis P: Proteochemometrics: A tool for modelling the molecular interaction space. In Chemogenomics in drug discovery – a medicinal chemistry perspective. Weinheim: WileyVCH; 2004:289–309.
 5.
Brereton RG: Chemometrics: Data Analysis for the Laboratory and Chemical Plan. John Wiley & Sons; 2003.
 6.
Roos H, Karlsson R, Nilshans H, Persson A: Thermodynamic analysis of protein interactions with biosensor technology. J Mol Recognit 1998, 11: 204–210. 10.1002/(SICI)10991352(199812)11:1/6<204::AIDJMR424>3.0.CO;2T
 7.
Andersson K, Gulich S, Hamalainen M, Nygren PA, Hober S, Malmqvist M: Kinetic characterization of the interaction of the Zfragment of protein A with mouseIgG3 in a volume in chemical space. Proteins 1999, 37: 494–498. 10.1002/(SICI)10970134(19991115)37:3<494::AIDPROT16>3.0.CO;2F
 8.
Andersson K, Choulier L, Hämäläinen MD, Van Regenmortel MH, Altschuh D, Malmqvist M: Predicting the kinetics of peptideantibody interactions using a multivariate experimental design of sequence and chemical space. J Mol Recognit 2001, 14: 62–71. 10.1002/10991352(200101/02)14:1<62::AIDJMR520>3.0.CO;2T
 9.
Freyhult EK, Andersson K, Gustafsson MG: Structural Modeling Extends QSAR Analysis of AntibodyLysozyme Interactions to 3DQSAR. Biophys J 2003, 84: 2264–2272.
 10.
Hoerl A, Kennard R: Ridge Regression: biased estimation for nonorthogonal problems. Technomoetrics 1970, 12: 55–67.
 11.
Geladi P, Kowalski B: Partial leastsquares regression: A tutorial. Anal Chim Acta 1986, 185: 1–17. 10.1016/00032670(86)800289
 12.
Höskuldsson A: PLS regression methods. J Chemom 1988, 2: 211–228.
 13.
Gustafsson MG: A probabilistic derivation of the partial leastsquares algorithm. J Chem Inf Comput Sci 2001, 41: 288–294. 10.1021/ci0003909
 14.
Golbraikh A, Trophsa A: Beware of q^{2}! J Mol Graph Model 2002, 20(4):269–276. 10.1016/S10933263(01)001231
 15.
Wold S: Validation of QSAR's. Quant Struct Act Relat 1991, 310: 191–193.
 16.
Ortiz AR, Pisabarro MT, Gago F, Wade RC: Prediction of drug binding affinities by comparative binding energy analysis. J Med Chem 1995, 38: 2681–2691. 10.1021/jm00014a020
 17.
Baroni M, Costantino G, Cruciani G, Riganelli D, Valigi R, Clementi S: Generating Optimal Linear PLS Estimations (GOLPE): An Advanced Chemometric Tool for Handling 3DQSAR Problems. Quant Struct – Act Relat 1993, 12: 9–20.
 18.
Ortiz A, Pator M, Palomer A, Cruciani G, Gago F, Wade R: Reliability of Comparative Molecular Field Analysis Models: Effects of Data Scaling and Variable Selection Using a Set of Human Synovial Fluid Phospholipase A2 Inhibitors. J Med Chem 1997, 40: 1136–1148. 10.1021/jm9601617
 19.
Cho S, Tropsha A: CrossValidated R2Guided Region Selection for Comparative Molecular Field Analysis: A Simple Method To Achieve Consistent Results. J Med Chem 1995, 38: 1060–1066. 10.1021/jm00007a003
 20.
Hoffman B, Cho S, Zheng W, Wyrick S, Nichols D, Mailman R, Tropsha A: Quantitative structureactivity relationship modeling of dopamine D(1) antagonists using comparative molecular field analysis, genetic algorithmspartial leastsquares, and K nearest neighbor methods. J Med Chem 1999, 42(17):3217–26. 10.1021/jm980415j
 21.
Höskuldsson A: Variable and subset selection in PLS regression. Chemometrics and Intelligent Laboratory Systems 2001, 55: 23–38. 10.1016/S01697439(00)001131
 22.
Ambroise C, McLachlan GJ: Selection bias in gene extraction on the basis of microarray geneexpression data. Proc Natl Acad Sci USA 2002, 99: 6562–6566. 10.1073/pnas.102102699
 23.
Lapinsh M, Prusis P, Gutcaits A, Lundstedt T, Wikberg JE: Development of proteochemometrics: a novel technology for the analysis of drugreceptor interactions. Biochim Biophys Acta 2001, 1525: 180–190.
 24.
Hamaguchi N, True T, Goetz A, Stouffer M, Lybrand T, Jeffs P: Alpha 1adreneric receptor subtype determinants for 4piperidyl oxazole antagonists. Biochemistry 1998, 37: 5730–5737. 10.1021/bi972733a
 25.
Eriksson L, Johansson E, KettanehWold N, Wold S: Introduction to Multi and Magavariate Data Analysis using Projection Methods (PCA & PLS). Umetrics, Umeå, Sweden; 1999.
 26.
Pastor M, Cruciani G, McLay I, Pickett S, Clementi S: GRidINdependent descriptors (GRIND): a novel class of alignmentindependent threedimensional molecular descriptors. J Med Chem 2000, 43: 3233–3243. 10.1021/jm000941m
Acknowledgements
This work was supported by the Swedish Research Council (62120012083, 62120024711), Carl Tryggers stiftelse (Stockholm), the Göran Gustafsson foundation (Stockholm), and the faculty of science and technology (Uppsala University).
Author information
Additional information
Authors' contributions
E.F. and M.G. devised and implemented the proposed double CV loop procedure, the feature selection algorithms, and a numerically efficient version of ridge regression required. P.P., M.L., J.E.S.W. provided the data sets studied together with experience and insights gained from their earlier work on proteochemometrics. V.M. and M.G. supervised the project. All authors read and approved the final manuscript.
Rights and permissions
About this article
Received
Accepted
Published
DOI
Keywords
 Partial Little Square
 Performance Estimate
 Partial Little Square Regression
 Ridge Regression
 Cross Term