A machine learning approach for predicting methionine oxidation sites

Abstract

Background

The oxidation of protein-bound methionine to form methionine sulfoxide, has traditionally been regarded as an oxidative damage. However, recent evidences support the view of this reversible reaction as a regulatory post-translational modification. The perception that methionine sulfoxidation may provide a mechanism to the redox regulation of a wide range of cellular processes, has stimulated some proteomic studies. However, these experimental approaches are expensive and time-consuming. Therefore, computational methods designed to predict methionine oxidation sites are an attractive alternative. As a first approach to this matter, we have developed models based on random forests, support vector machines and neural networks, aimed at accurate prediction of sites of methionine oxidation.

Results

Starting from published proteomic data regarding oxidized methionines, we created a hand-curated dataset formed by 113 unique polypeptides of known structure, containing 975 methionyl residues, 122 of which were oxidation-prone (positive dataset) and 853 were oxidation-resistant (negative dataset). We use a machine learning approach to generate predictive models from these datasets. Among the multiple features used in the classification task, some of them contributed substantially to the performance of the predictive models. Thus, (i) the solvent accessible area of the methionine residue, (ii) the number of residues between the analyzed methionine and the next methionine found towards the N-terminus and (iii) the spatial distance between the atom of sulfur from the analyzed methionine and the closest aromatic residue, were among the most relevant features. Compared to the other classifiers we also evaluated, random forests provided the best performance, with accuracy, sensitivity and specificity of 0.7468±0.0567, 0.6817±0.0982 and 0.7557±0.0721, respectively (mean ± standard deviation).

Conclusions

We present the first predictive models aimed to computationally detect methionine sites that may become oxidized in vivo in response to oxidative signals. These models provide insights into the structural context in which a methionine residue become either oxidation-resistant or oxidation-prone. Furthermore, these models should be useful in prioritizing methinonyl residues for further studies to determine their potential as regulatory post-translational modification sites.

Background

Reactive oxygen species (ROS) are well known for their harmful effect on cellular constituents [1]. However, a more nuanced view has emerged during the last years. It is now clear that certain ROS, including H₂O₂, can function as messengers [2]. To act as an effective messenger, hydrogen peroxide has to bring about a reversible change in the activity of a protein through post-translational modification (PTM). The amino acids that are used as PTM sites often have a functional group that is able to act as a nucleophile during the modification reaction. In this regard, the sulfur contained in the side chain of cysteine and methionine is liable to be oxidized by H₂O₂. Under mild oxidative conditions, cysteine forms cystine through a disulfide bridge, while methionine is preferentially oxidized to methionine sulfoxide. Both oxidation reactions can be reverted through reduction reactions catalyzed by enzymes. Disulfides are reduced back to the thiol form by various reductases [3]. On the other hand, MetO is reduced back to methionine by the enzyme methionine sulfoxide reductase (Msr), present in most aerobic cells [4].

Like phosphorylation of serine, sulfoxidation of methionine is a reversible covalent modification capable of modifying the physicochemical properties of the complete protein, which, in turn, can affect the stability and/or activity of the target protein [5, 6]. Indeed, it has been demonstrated that sulfoxidation of specific methionine residues can lead to both activation [7–9] and inactivation [10, 11] of the modified protein. Moreover, the oxidation of specific methionine sites may also impact the function of a protein in an indirect manner, by facilitating or hindering the occurrence of other functional PTM such as phosphorylation of nearby serine residues [12–14].

The perception that methionine sulfoxidation may provide a mechanism to the redox regulation of a wide range of cellular processes, has stimulated some proteomic studies [15–17]. This proteomic approach, despite the technical difficulties involved in the discrimination between physiological and artifactual modifications, has allowed to identify a considerable number of cellular proteins as possible targets of oxidative signals. Furthermore, these proteomic efforts have allowed to pinpoint the sites of oxidation over the target proteins. Nevertheless, these experimental approaches, besides being expensive, are labor-intensive and time-consuming. In view of this, it is highly desirable to develop in silico methods aimed to predict methionine oxidation sites. Indeed, in the field of protein phosphorylation, the prediction of phosphorylation sites using computational tools has attracted considerable research attention [18–20]. Unfortunately, computational approaches to predict methionine oxidation sites have garnered much less attention, and only very recently some efforts have been devoted to this purpose [21].

Herein, we describe predictive models based on computational intelligence, aimed at accurate prediction of methionine sulfoxidation sites.

Results

For each methionine residue from the training dataset, a total of 76 characteristics were evaluated as described in the “Methods” section. 52 of these characteristics were derived from the primary structure while the remaining 24 characteristics were related to the tertiary structure. These collections of features will be referred to as, Whole, Primary and Tertiary, respectively. Using these different sets of characteristics, we designed a number of machine learning (ML) predictive models, namely random forests (RF) [22], support vector machines (SVM) [23] and neural networks (NN) [24], which were intensively tested in a comparative approach. The results obtained from these comparative studies are presented in the following subsections.

Predicting methionine oxidation with random forest

The performance of various RF-based models was evaluated in terms of the area under the ROC curve (AUC), accuracy, sensitivity, specificity, F-measure and MCC (Matthews Correlation Coefficient). The results obtained using different subsets of characteristics, for both training and testing datasets, are shown in Table 1 (first four rows of “TRAINING SET” and “TESTING SET” sub-tables from Table 1). In addition to the above described subsets of characteristics, we also used a subset formed by the most relevant features (see “Methods” section). To this end, the characteristics were ranked using the maximum relevance minimum redundancy (mRMR) method [25], which uses a ranking criterion based on the trade-off between the relevance to the output (oxidable) and the redundancy between the input characteristics. In this way, a final subset of 54 features was identified as the optimal (giving the maximum AUC) feature set (see “Methods” section for details).

Table 1 Performance rates with three different ML models

Comparison with other machine learning models

To account for the potential of RF as an effective ML approach to predict the oxidation of methionine, we have compared it with two other classical ML models: SVM and NN (see “Methods” section). The performance of these alternative methods is also summarized in Table 1. These results showed differences in favor of RF, with respect to SVM and NN, as RF gave high AUC and accuracy rates with a better balance between sensitivity and specificity rates for data from the testing set.

However, as those results in Table 1 correspond to single ML models applied on a same training/testing set, a more comprehensive evaluation of each ML-model’s predictive potential was needed. In this vein, Table 2 and Fig. 1 show the results from a bootstrapping strategy: for each ML model and feature subset (Primary, Tertiary, Whole and mRMR), 100 bootstrap re-samples were generated and 10-fold cross-validation (with 5 repetitions) were used to train and fit each model. Mean performance rates and standard deviation on the training and testing sets (after ROC’s cut-off probability adjusting on the evaluation sets) are shown in Table 2. The best overall results on the testing sets (high accuracy rate with balanced sensitivity and specificity) were obtained with RFs, showing significant differences with respect to SVMs and NNs (see t-test p-values in Table 3). Remarkably, very similar results were obtained with both the mRMR subset and the whole set of 76 characteristics. In general, SVMs and NNs showed similar efficacy rates, with accuracy numbers that were lower than those given by the RFs and worse balances between sensitivity specificity rates (see Table 3 and Fig. 1).

Fig. 1

Performance rates distributions for bootstrapping resamples. Box-plots of the performance rates on the testing sets (after ROC’s cut-off probability adjustment on the evaluation sets) for bootstrapping resamples. Data set mRMR 54 features. Number of resamples = 100

Table 2 Performance rates for three different ML approaches: mean (sd)

Table 3 Models comparison. T-test p-value from bootstrap results on the testing sets

The quantification of the predictive importance of each variable is a key factor to interpret data and to understand the phenomena underlying methionine oxidation. Thus, we resorted to the Gini-index importance to assess the relevance of the variable used for the RF classifiers as input characteristic. Fig. 2 shows the 20 most relevant variables as estimated by the RF on the training set (100 bootstrap resampling), along with the distribution (box-plot) of their averaged decrease in Gini-index (see “Methods” section). As it can be observed, the accessibility to the solvent, the proximity to other methionyl residues and the distance to the closest aromatic residue are among the variables with the highest predictive importance (Fig. 2).

Fig. 2

Variable Importance. Box-plots of the GI of the 20 most relevant predictors for the RF classifier. From top to bottom: variables in decreasing order of average GI (100 bootstrapping resamples). Dataset: mRMR 54 features

Discussion

Protein-bound methionine is readily oxidized to methionine sulfoxide, which can drastically affect the biological activity of the modified proteins. Although this fact has been known for many years now, our perception of the functional implication of methionine sulfoxidation has evolved over time. Initially, this chemical modification was detected in proteins that had been purified from tissues following laborious experimental procedures. Hence, there was a reasonable doubt of whether the observed modification was present in the natural tissues, or whether it arose from some oxidation during the manipulations in vitro [26]. A decade later, it was clear that the oxidation of methionine in proteins takes place in vivo [27]. However, the presence of methionine sulfoxide in proteins was considered just as an inevitable and harmful consequence of oxidative stress. Later on, the regard of methionine oxidation as mere oxidative damage would give pass to a more benign judgment.

The finding that oxidation of protein-bond methionine residues to methionine sulfoxide is one of the few protein oxidation events that are reversible in vivo, led to the appealing hypothesis of methionine residues as endogenous antioxidants in proteins [28]. Indeed, reversible oxidation/reduction of methionine residues in proteins can serve as a scavenger system to remove ROS, and the importance of methionine oxidation in the antioxidation defense has gathered strong experimental evidences since then [29, 30]. On the other hand, although ROS have traditionally been thought as harmful by-products of respiratory metabolism, that notion has slowly given way to a more nuanced view of ROS as important signaling molecules [1]. In this context, a new functional role for methionine modification can be envisioned. Methionines that undergo sulfoxidation may serve as PTM sites fulfilling a signaling role, acting as on/off sensors of oxidative stress in certain proteins. A number of such proteins has already been identified [31–34].

Our current awareness of the functional relevance of methionine oxidation at certain sites, demands tools for the prediction of such sites. As a first step towards this goal, in this study we have developed machine learning models for predicting whether a given methionine residue would be oxidized in vivo after an oxidative challenge. In the past, driven by the interest to expand the shelf life of therapeutic proteins, considerable effort has been devoted to predict the reactivity in vitro of methionine residues towards oxidants, using for this purpose molecular modeling [35]. However, because of the limited number of proteins analyzed and the nature of the data used (obtained from in vitro kinetic assays) the use of these molecular models cannot be extrapolated to a more general framework of methionine oxidation prediction. In contrast, herein we have used a large collection of data encompassing over hundred proteins containing 122 methionyl residues that have been empirically detected as methionine sulfoxide. The fact that these sulfoxidized methionines are present within the cells, means that the proteome data used in the current study represents a steady-state situation, in which oxidation after hydrogen peroxide challenge is balanced by reduction catalyzed by methionine sulfoxide reductases. Therefore, our study is, to the best of our knowledge, the first attempt to train and test computational models aimed to predict the oxidation status of protein-bound methionines, when such protein are found into their subcellular environment.

In this work, we have used machine learning models to predict the oxidation of methionine in protein sequences. To this end, all the models we have been dealing with handled two output classes: modified and unmodified methionine sites, where the negative category (non-oxidized methionine) is defined by the absence of the modification. It may be possible that some of the methionine sites labeled as negative would be actually modified sites, but the experimental procedure failed to detect them? Although such a possibility never can be fully ruled out, it seems unlikely. Indeed, protein abundance is a major factor for the detection of PTMs by mass spectrometry. To this respect, an important characteristic of our ML approach is that each methionine site belonging to the negative dataset had its own internal control. Since negative methionines were obtained from proteins containing at least one positive methionine, we can be confident that the non-oxidized methionine was present at equimolar concentration with respect to other methionine detected as MetO during the same experiment. Nevertheless, a caveat that should be taken into consideration is that the whole dataset come from a single proteomic study using Jurkat cells [15]. Whether the cellular processes taken place in this cell line represent those operating in animal tissues, is an issue that remains to be solved [36]. In any event, future effort directed to identify new methionine sulfoxidation sites in vivo, using different species, tissues and experimental conditions, will lead to improved predictive models.

The unbalanced distribution of the output classes (oxidized vs non-oxidized) and the proportion of missing data in the dataset are two characteristics that deserve discussion because they affect the performance of the predictive models. The former has to do with the severe class imbalance (the positive dataset only represents 12.5% of the whole dataset). When training and tuning the predictive models, we had to deal with this unwanted issue. Fortunately, the unbalanced distribution problem could be resolved using sampling techniques or ROC curve post-processing approaches. On the other hand, missing data can dramatically affect the effectiveness of the classifiers if not appropriately treated. Moreover, the predictive models used in the current study cannot deal with missing values, which make missing data imputation unavoidable. Three different missing data imputation methods have been tested in our study, k-nearest neighbors (KNN) imputation, median imputation and bagging imputation [37]. KNN imputation was carried out by finding the k closest samples (Euclidean distance) in the training set. Imputation via medians takes the median of each predictor in the training set and used them to fill missing values. This method is simple and fast, but treats each predictor independently and may be inaccurate. Imputation via bagging fits a bagged tree model for each predictor (as a function of all the other features). This method, which is simple and accurate, gave us the best results in our study although it had higher computational cost.

Since protein sequences are easily determined and easy to work with, initially we resorted to features that could be extracted using only protein sequence information, to build the so-called Primary models. Despite the limitation of disregard valuable 3D structural information, these models performed modestly well (Table 2), with balanced sensitivity and specificity in spite of a remarkable imbalance between the total numbers of oxidized and non-oxidized methionine sites in the training samples. Nevertheless, when features related to the spatial structure of the protein were included into the models, their performance improved substantially. This finding is consistent with previous studies demonstrating the importance of structural variables (such as the solvent accessible area of the methionine and its spatial proximity to aromatic residues) in determining the oxidation state of methionyl residues in the proteins within living cells [38]. Interestingly, the use of computational techniques to filter features on the base of their high relevance and low redundancy (mRMR), allowed us to conclude that a reduced number of features (54 out of 76) was enough to obtain the best results.

With respect to the different ML approaches tested herein, the best performance was obtained using RF, while SVM and NN behave worse in general when compared to RF (Table 2 and Fig. 1). There is not a clear reason why this should be that way. However, again the heterogeneous nature of the data, including the intrinsically unbalanced distribution of the output classes, make the RF a better ML approach for this particular problem of methionine oxidation site prediction. The “ensemble nature” of RF (a large pool of decision trees is built during the training phase) does its best to deal with the challenge of predicting new input patterns as those found in the testing sets, thus giving high performance rates while the balance between sensitivity and specificity remains. Nevertheless, since the limitation of available data and the unbalanced characteristic of the dataset may affect the performance of the classifier, further work for refining and improving the prediction model will be carried out using additional classification methods and additional dataset when they become available. We also provide a stand-alone program based on the RF model described herein. This software can be downloaded from google.drive.scripts, where any interested user will also find detailed use instructions.

Phosphorylation is the most common post-translational modification [39]. Many of the cellular responses triggered by oxidative stress are known to be mediated, at some point, by signaling cascades involving protein phosphorylation [40, 41]. Recent studies have suggested that the crosstalk between serine/threonine phosphorylation and methionine sulfoxidation may serve to fine-tune the cellular response to oxidative signals [12, 14]. In line with these previous works, we have observed that including features related to phosphorylation information (see Methods for details) in the predictive model of methionine oxidation does contribute to its performance (see the list of relevant features filtered by the mRMR algoritm, as well as Fig. 2). All in all, these works point to a relevant role for methionine oxidation in the regulation of protein function.

Conclusions

In this study we have designed and tested computational models to predict methionine oxidation sites. High accuracy rates as well as balanced specificity and sensitivity values were obtained. The best performances were obtained when random forests were used, while neural networks and support vector machines behaved less effectively, in general.

From the 76 features used in the design of our predictive models, some variables related to the protein structure, such as solvent accessibility (SASA) and the proximity of aromatic residues, have been identify among those making the highest contribution to the predictive power of the random forest classifier. Some characteristics regarding phosphorylation, such as the distance to the closest phosphorylable residue, have also been detected as relevant features. This fact supports the hypothesis of methionine sulfoxidation playing an important role in the crosstalk with protein phosphorylation.

As our understanding of the role played by methionine sulfoxidation in all aspects of cellular biology continues to expand, these computantional predictive models will become increasingly valuable, especially in hypothesis-driven investigations. Moreover, the availability of reliable predictive tools should stimulate further investigations aimed to gain a better understanding of the interplay between sulfoxidation and phosphorylation during cellular redox signaling.

Methods

Datasets

Data regarding methionine residues detected as methionine sulfoxide in vivo were taken from reference [15]. This set was further curated to exclude protein entries that did not contain at least one methionine showing a degree of oxidation, as defined in [15], equal or greater than 20%. Using PDB cross-references from UniProt (http://www.uniprot.org), this collection was further constrained to those proteins with known structure. In general, since many proteins were homooligomers, most crystal structures yielded a large number of duplicated observations, which were searched for and eliminated using a R script. Eventually, after removing redundancy and filtering out low quality structures (for instance, those where the target methionine did not appear resolved), we assembled a collection of 113 unique polypeptides of known structure, containing 975 methyonil residues, 122 of which were oxidation-prone (positive dataset) and 853 were oxidation-resistant (negative dataset).

Feature extraction

For each methionine residue from the dataset described above, a total number of 76 features were extracted. These features included 20 variables of the type NT_X, defined as the number of positions in the protein sequence from the analysed methionine to the closest X residue toward the N-terminus, where X belong to the set of 20 proteinogenic amino acids. Similarly, other 20 features of the type CT_X were assessed, in this occasion, counting towards the C-terminus.

Four additional features were related to the conservation of the considered methionine during evolution. To assess these features, besides the human sequence, the orthologous proteins from Pan troglodytes, Gorilla gorilla, Rattus norvegicus, Bos taurus, Gallus gallus, Xenopus tropicalis and Danio rerio were aligned. These alignments were used to compute the Shannon entropy according to the equation:

$$ entropy = - \sum_{i = 1}^{21} f_{i} log_{21}(f_{i}), $$

where f _i is the relative frequency of the symbol i at the analysed position across the alignment. Thus, for instance, f _M stands for the relative frequency of methionine. The logarithmic base was taken 21 because in addition to the 20 proteinogenic amino acids, the symbol ‘-’ was considered when indels were present. For each analysed methionine, the variables mean.entropy and sd.entropy were computed as the mean and standard deviation, respectively, of the entropy determined at all the positions of the corresponding protein.

Eight further features related with PTM sites were evaluated. Concretely, the variables Met2S, Met2T and Met2Y inform about the distance, in the primary structure, between the analysed methionine and the closest serine, threonine and tyrosine phospho-acceptor, respectively. It should be noted that

$$Met2X = \text{min}(NT\_X, CT\_X). $$

On the other hand, Met2S_PTM, Met2T_PTM and Met2Y_PTM collect the distances to the closest corresponding phosphosites. That is, to the closest phospho-acceptor that has been shown to be phosphorylatable [42]. The other two PTM-based features were closer10res, defined as the number of phosphorylatable residues in a radius of 10 amino acids from the analysed methionine, and away.ptm calculated according to the following expression:

$$away.ptm = \underset{X \in \{S, T, Y\}}{\text{min}}(Met2X\_PTM). $$

The 52 features described hitherto can be extracted from the primary structure of the involved proteins. However, to compute the 24 features that we will introduce next, information about the 3D structure of the protein was essential.

Thus, we defined and computed four new variables related to PTM sites. The first of these variables, referred to as closest.ptm.chain, gives the distance in ångströms between the considered methionine and the closest phosphorylatable residue (either Ser, Thr or Tyr experimentally shown to be phosphorylated) present in the same polypeptide chain that the methionyl residue. If we remove the constraint of both sites having to be intrachain, then we will be dealing with the feature closest.ptm.pdb. The feature closer10A.chain provides the number of phosphorylatable sites, found on the same polypeptide chain, within a sphere of radius 10Å centred at the relevant methionine. Analogously, closer10A.pdb gives the number of phosphorylatable sites within the sphere, regardless of the chain hosting them.

In a recent work we reported that methionyl residues forming part of an S-aromatic motif are less prone to be oxidized [38]. Therefore, 16 additional features related to this non covalent bond were used. Concretely, Xd.chain was defined as the distance in ångströms between the sulfur atom from the analysed methionine and the nearest X aromatic residue within the same polypeptide chain, being X either Y (Tyr), F (Phe) or W (Trp). If the aromatic residue is allowed to be in a different polypeptide molecule, we refer to this feature as Xd.pdb. The variables nX.chain and nX.pdb inform about the number of X aromatic residues (within the same polypeptide molecule or not, respectively) at a distance < 7Å from the methionine. The feature numberBonds.chain was computed according to:

$$numberBonds.chain = \sum_{X \in \{Y, F, W\}} nX.chain. $$

Similarly, numberBonds.pdb was defined as:

$$numberBonds.pdb = \sum_{X \in \{Y, F, W\}} nX.pdb. $$

In addition, the variables closestAro.chain and closestAro.pdb were computed as:

$$\begin{array}{*{20}l} closestAro.chain &= \underset{X \in \{Y, F, W\}}{\text{min}} (Xd.chain),\\ closestAro.pdb &= \underset{X \in \{Y, F, W\}}{\text{min}} (Xd.pdb). \end{array} $$

Other two features, SASA.chain and SASA.pdb, were related to the solvent accessible surface area of the methionine residue. These variables were assessed with the program DSSP [43] and either the atomic coordinates of the single polypeptide chain harboring the methionine (for SASA.chain), or the atomic coordinates of the whole protein (for SASA.pdb).

The B factor of the sulfur atom from the methionine of interest extracted from the PDB file used was recorded in the variable Bfactor.

Finally, dpx measures the depth of the sulfur atom from the considered methionine, defined as the distance in ångströms between the S atom and the closest atom from the protein exposed to the solvent [44].

The data file with all these extracted features used in our study is available at github.data

Machine learning methods

In the current study we used RFs to design predictive models of methionine oxidation sites. RFs are ensemble machine learning methods for classification, that function by constructing a large pool of decision trees during the training phase. The final output will be the mode of the classes given by the individual trees in the pool. The method combines Breiman’s ‘bagging’ idea and the random selection of features (i.e. predictor-set split) in order to construct a collection of decision trees with controlled variation [22].

The quantification of the predictive importance of each variable was carried out by means of the Gini-index Importance (GI). The Gini-index [45] for a given node of a decision tree can be defined as

$$p_{1} (1 - p_{1}) + p_{2} (1 - p_{2}), $$

where p ₁ and p ₂ are the “class 1” and “class 2” probabilities, respectively. For a binary-classification problem, p ₁+p ₂=1 and the previous equation could be written as 2p ₁ p ₂. The Gini-index minimizes when either p ₁ or p ₂ drives towards zero, and maximizes when p ₁=p ₂, i.e. when the node is “least pure”. The GI uses the decrease of Gini-index (impurity) after a node split as a measure of variable relevance. The average decrease in Gini-index over all trees in the RF defines the GI.

In general, when it comes to predictive performance, there are cases where SVMs do better than RFs, and vice versa. The same is true for NNs with respect to other ML approaches. Thus, for comparative purposes we also developed classifiers based on SVM [23], as well as on NNs [24].

Model tuning

For RF model-fitting in our experiments regarding methionine oxidation, the only sensible tuning hyper-parameter would be the number of variables (predictors) randomly sampled as candidates at each split (usually known as mtry). We fixed the value of this parameter at the optimal recommended value $\lfloor \sqrt {number\ of\ predictors}\rfloor$ [22, 46]. On the other hand, the number of trees to grow was fixed to 1000 to ensure that every input pattern could be predicted at least a few times [47].

For SVMs, a Gaussian radial basis function (RBF) kernel $k(x, x^{\prime }) = e^{-\sigma ||x - x^{\prime }||^{2}}$ was used (being k a function that calculates the inner product 〈Φ(x),Φ(x ^′)〉 of two vectors x, x ^′ for a given projection Φ:X→H). The problem of model selection (parameter tuning) was partially addressed by an empirical observation for the Gaussian RBF kernel, where the optimal values of the hyper-parameter σ are known to lie in between the 0.1 and 0.9 quantile of the ||x−x ^′|| statistics [48, 49]. Thus, a sample of the training set was used to estimate these quantiles. Any value of σ comprised within the quantile interval results in good performance. In this way, the σ parameter was automatically estimated. Additionally, the optimal hyper-parameter cost, that represents the cost of constraints violation and stands for the ‘C’-constant of the regularisation term in the Lagrange formulation, was tuned as the one of 12 incremental values in $\{2^{i}\}_{i=-2}^{9}$ that optimises the area under the ROC curve (AUC) of the SVM classifier.

Fully connected single-hidden-layer feed-forward NNs—Multilayer Perceptrons (MLP) [50]—were also constructed and trained with different combinations of parameters to search for the best performance rates in the prediction of methionine oxidation. Optimisation of the NNs was done via the error back-propagation algorithm [50]. The network size (i.e., number of hidden units in the single hidden layer) and weight decay were the tuned parameters, selecting the combination of values that provided the highest AUC. All the trained MLPs had a number of outputs that was equal to the number of classes (i.e. n=2), with logistic activation function for all the hidden and output neurons. Weights were randomly initialised, and maximum number of epochs was fixed to 100 [51].

For each predictive model, the best values for the fitted parameters are computed as those giving the highest averaged AUC via 10-fold cross-validation on the training dataset (in Table 4 the best hyper-parameters for each ML model in Table 1 are shown).

Table 4 Model tuning. Best hyper-parameters

Resampling methods for model fitting

The data set was divided into three independent sets, 80% (98 ‘positive’; 683 ‘control’) patterns for training, 6.7% (8 ‘positive’; 57 ‘control’) patterns for evaluation (these pattern set is used to compute the optimal threshold for the ROC curves) and, finally, 13.3% (16 ‘positive’; 113 ‘control’) for testing. To preserve the unbalanced nature of the original class distribution within the splits, a stratified random sampling strategy was used. To estimate the efficacy of the prediction model across the training set, six performance measures—AUC, accuracy, sensitivity, specificity, F-measure and Mathews-Correlation-Coefficient (MCC)—of the out-of-bag (OOB) samples for 10-fold cross-validation with 5 repetitions (50 re-samplings) were calculated and the mean and standard deviation of those rates were computed. To compute the latter five performance measures, and given the following general table for any binary classification problem (with two classes: Yes/No),

where TP, FP, TN and FN stand for true positive, false positive, true negative and false negative, respectively, we have used the following well-known formulae:

• accuracy =(T P+T N)/(T P+T N+F P+F N),

• sensitivity =T P/(T P+F N),

• specificity =T N/(T N+F P),

• F-measure $= 2 \frac {precision \times sensitivity}{precision + sensitivity}$, where p r e c i s i o n=T P/(T P+F P),

• MCC $= \frac {TP \times TN - FP \times FN}{\sqrt {(TP + FP) (TP + FN) (TN + FP) (TN + FN)}}$.

With respect to the two last performance measures, i.e. the F-measure and MCC, although both of them have been included in our analyses because they both are usually used in machine learning as measures of the quality of binary classifications, the F-measure has to be taken with caution, as it does not take the true negatives into account. For this reason, and given that our dataset is seriously unbalanced towards the negative samples, the MCC may be preferable to assess the performance of our binary classifiers.

The entire training set was used to fit a final model and its performance was finally measured on the testing set. For bootstrap resampling (see “Results” section), 100 random resamples were generated and 10-fold cross-validation (with 5 repetitions) was used to train and fit each model (RF, SVM and NN). The caret R package [52, 53] (R version 3.3.3) has been used for model fitting with SVM (package kernlab [49]), NN (package RSNNS [51]) and RF (package randomForest [47]).

One of the more severe circumstances that can dramatically affect the effectiveness of prediction models is class imbalance, i.e. the unbalanced relative frequency of one class in the training set as compared to the other class. In our study, class imbalance is inherent to the procedure being followed for data acquisition (see “Datasets” section): of the complete set of methionine residues found in the 113 polypeptides analysed, only 122 out of 975 appeared as oxidised, i.e. a mere 12.5%. This can result in predictive models that can easily get high accuracy rates at the expense of unacceptable sensitivity figures. For example, the most ‘naïve’ predictive model consisting in classifying all methionine residues as ‘non oxidised’ would give 87.5% accuracy and 100% specificity, but an unwelcome 0% sensitivity. To further characterize this sensitivity issue, we launched a pool of 1000 “random predictions” over the entire dataset. For each of these random predictions, 12.5% of the 975 patterns were randomly chosen as oxidized sites. In this way, the mean accuracy (78.1%) and specificity (87.5%) were high enough, but, as expected, the mean sensitivity was unacceptably low, 12.4% (standard deviation 0.0071, 0.0283 and 0.0040, respectively).

To counteract the negative effects of class imbalance, different approaches have been proposed in the literature [37]. These approaches include model tuning (using metrics alternative to accuracy such as ROC, Cohen’s Kappa or sensitivity), adjusting of prior probabilities, cost-sensitive training, use of alternative ROC-curve cutoffs, or use of specific sampling methods. In the current study a combination of the two latter gave the best results. Prior to model tuning and fitting, we used the down-sampling technique to get a more balanced training dataset. The general idea of this method is to artificially down-sample the majority class (i.e. ‘non oxidised’ class).

On the other hand, after model training using this down-sampled set of patterns, we used the ROC curve to determine alternative cutoffs for the probabilities predicted by the model. Using this ROC curve, an appropriate balance between sensitivity and specificity can be determined. Although several techniques do exist for determining a new cutoff, the more general approach is to find the point on the ROC curve that is closest (i.e., the shortest distance) to the perfect model (with 100% sensitivity and 100% specificity), which is associated with the upper left corner of the plot [4]. To determine this cutoff point without biasing the results obtained from the final testing dataset, an independent evaluation dataset was used (see above). In Fig. 3 the ROC curves obtained from the RF, SVM and NN classifiers (corresponding to the performance results in the last row of each model’s data in Table 1) on the evaluation dataset is shown together with the computed alternative cutoff. As it can be observed in the figure, the alternative cutoff gives a better balance between sensitivity and specificity. However, as it can be observed in Table 5, this better balance between sensibility and specificity is obtained at the expense of accuracy. For comparison purposes, in Table 5 those performance results (on the testing set) from the RF model of Table 1 are shown again (computed alternative cutoff: 0.392), together with the results for this same model but with the standard cutoff of 0.5.

Fig. 3

ROC curves. From top to bottom: ROC curves and AUC values computed on the evaluation patterns for the RF, SVM and NN models, respectively. The point in each curve that gives the best balance between sensitivity and specificity rates has been marked and annotated with the corresponding “alternative” threshold and efficacy values. Solid black box: AUC = 1 reference area. Dashed gray line: smoothed ROC curve. Solid gray line: random guess

Table 5 Performance rates for RF with two alternative ROC cutoffs

Feature selection with the mRMR method

We used the minimum redundancy maximum relevance (mRMR) method [25] to rank the importance of the 76 features, based on the trade-off between the relevance to the output (oxidable) and the redundancy between the input characteristics. This method is based on the concept of mutual information. Given two variables, x and y, their mutual information can be defined as

$$I(x,y) = \int \int p(x, y) log \frac{p(x,y)}{p(x)p(y)} dx dy. $$

When the goal is to select N features from the whole feature set (Ω), according to their minimum redundancy (among them) and maximum relevance (with respect to the target or output, o), the first feature added to this set of selected characteristics, Ω _s , is selected according to the concept of maximum relevancy. That is, the feature f with the highest I(f,o). The rest of features are selected in an incremental way: earlier selected features remain in the feature set Ω _s . Suppose m features have been already selected, and we want to select an additional feature from the set Ω _p =Ω−Ω _s

The next characteristic f _j ∈Ω _p to be selected, i.e. to be included in Ω _s , is the one that maximises the mRMR function, given by

$$max_{f_{j} \in \Omega_{p}}\left[ I\left(\, f_{j}, o\right) - \frac{1}{m}\sum_{f_{i} \in \Omega_{s}}I\left(\, f_{j}, f_{i}\right)\right] $$

To determine the final set of N selected features, an incremental approach was followed: for each number of selected characteristics N=2,…76, a random forest was trained (with down-sampled patterns from the training set), its ROC’s cut-off probability was established using the evaluation set (see “Methods” section) and, finally the AUC for the testing set was measured. Following this strategy, a final set of N=54 features was identified as the optimal (maximum AUC) feature set.

The final set of 54 features, in order of selection by the mRMR algorithm, is the following (see Sec. Feature Extraction for a description of the characteristics): SASA.pdb, NT_M, away.ptm, CT_Q, CT_F, NT_R, NT_D, Met2Y, CT_L, nF.pdb, NT_I, Met2S, dpx, CT_G, NT_C, nY.chain, CT_D, NT_W, sd.entropy, CT_E, CT_H, CT_M, closestAro.chain, NT_K, CT_V, CT_A, closer10A.pdb, CT_R, CT_N, NT_A, NT_P, NT_N, CT_C, NT_L, SASA.chain, NT_E, CT_P, NT_H, CT_T, NT_F, NT_V, CT_K, NT_G, NT_T, Bfactor, nW.pdb, entropy, NT_Q, CT_S, NT_Y, CT_I, NT_S, Fd.pdb and CT_Y.