Prediction of conformational B-cell epitopes from 3D structures by random forests with a distance-based feature

Datasets

Datasets used in the studies are relevant to their goals and scopes. Some conformational epitope prediction models are constructed on the bound structures, while others are built on the unbound structures. Therefore, we use both bound and unbound dataset to evaluate and compare models.

We use the dataset published by Rubinstein as the benchmark bound dataset [26]. The bound dataset consists of 66 non-redundant Ag-Ab structures, available at: http://epitopia.tau.ac.il/trainData/.

We use the Liang's dataset as the benchmark unbound dataset [28]. Liang's dataset is compiled as follows: (1) 22 antigen-antibody complexes and their unbound structures were sourced from protein docking Benchmark 2.0 [37]; 59 representative antigen-antibody complexes were provided by [38]; 17 antigen-antibody complex structures were collected from [27]; (2) these structures were merged, and the complexes without available unbound structure were removed. Finally, a total of 48 complexes and their unbound structures were retained as the benchmark unbound dataset, available at: http://sysbio.unl.edu/services/.

In addition, the independent test set compiled from entries of the Conformational Epitope Database (CED) [39] is used, which contains 19 antigen structures with annotated epitopes. This dataset is available at: http://sysbio.unl.edu/services/.

We compile a benchmark dataset of 83 antigen sequences from Rubinstein's structure dataset, available at http://code.google.com/p/my-project-bpredictor/downloads/list. Hence, we can fairly compare the sequence-based models with structure-based models.

Epitope definition

There are several definitions ever used for the epitopes inferred from the X-ray structures of Ag-Ab complexes, such as the accessible surface area loss upon antibody binding or the distance between antigen residues and antibody residues. However, the study in [38] indicated that different epitope definitions are likely to give out similar results. Hence, we follow the commonly used distance-based definition. Specifically, an antigen residue separated from any antibody residue by a distance less than 4Å is defined as an epitope residue, and the distance between two residues is measured by the minimal Euclidean distance between the centers of any of their non-hydrogen atoms.

Thick surface patch

A residue is defined as the surface residue, if its relative accessible surface area (RASA) calculated by DSSP program [40] is more than 5%. When using the surface patch to describe the spatial characteristics of antigen residues, the epitope residues and non-epitope residues are considered to be distinct with respect to their surface patches. We notice that the surface patch only include the surface residues, therefore this raises a question: are the adjacent interior residues unimportant or unnecessary for the representation of spatial context? Clearly, the interior residues cannot be epitope residues, but it does not mean that they cannot influence surface residues, and the interior residues may contribute to the formation of epitope sites. In order to address the issue, the impact of interior residues cannot be neglected and should be investigated. In this study, we propose a new concept 'thick surface patch'. Formally, the thick surface patch of a surface residue is defined as a set of n nearest adjacent residues, including interior neighbors as well as surface neighbors. For simplicity, the thick surface patch and the surface patch are generally named 'residue patch' in the following sections.

Adjacent residue distance feature

The residue patch is critical for the conformational epitope prediction. However, contributions of residues in a patch may be distinct and depend on their distances to the central residue. Since existing methods usually used the patch of 20 residues, the analysis is implemented on the patch of this size.

To test whether the distances between adjacent residues and the central residue have impact on the state of the central residue, we calculate the average distance between adjacent residues and the central residue, and the average distance is compared between epitope patches and non-epitope patches for each central residue type. The results reveal that non-epitope patches have significantly less average distance than epitope patches (P = 1.59 × 10^-10 by paired t-test for the bound dataset, see Figure 1).

For further test, we compare the distance between k th nearest adjacent residues (k = 1, 2... 20) and the central residue in epitope patches versus non-epitope patches. The results show that the distance distribution in epitope patches is significantly different from that of non-epitope patches (P = 1.20 × 10^-7 by paired t-test for the bound dataset, see Figure 2).

According to the statistical analysis on the bound dataset, it is observed that the average distance of the patch and the distance distribution of the patch may help to distinguish the epitope patches from non-epitope patches. The similar conclusion can be drawn for the unbound dataset (data not shown).

Where x _i represents an adjacent residue in a patch, I = 1, 2, ...,n. $p_i=\frac{1}{d_i}$, and d _i is the Euclidean distance between x _i and the central residue (based on the nearest non-hydrogen atoms). In the feature, the contributions of adjacent residues in a patch are quantitatively represented and depend on their relative distances to the central residue.

Descriptors for residue patch

While constructing prediction models, each patch should be represented as a feature vector by using physicochemical and structural features. In addition to the adjacent residue distance feature, several popular physicochemical and structural features are used.

Relative accessible surface area: it is an important factor influencing the antigen-antibody binding, and the greater relative area of a surface residue means the greater probability of being an epitope residue. The relative accessible surface area of a residue is calculated by dividing its accessible surface area with the accessible surface area of fully exposed amino acid. The accessible surface areas of surface residues are calculated by using DSSP program [40], and the fully exposed amino acid area can be obtain from [41].

Here, M _ir is the value of residue type r at the sequence position i, according to the PSSM, and B _rr is the diagonal element of BLOSUM62 for residue type r. The same function is used in [28].

Secondary structure: secondary structures are proved as important factors for the Ag-Ab interaction, and epitopes are likely to have specific secondary structure elements versus non-epitope surfaces [42]. Here, we use DSSP to calculate the secondary structures of surface residues, and each secondary structure (helix, sheet or coil) is represented as a three-bit string, such as (1, 0, 0), (0, 1, 0) and (0, 0, 1), respectively.

Amino acid composition: amino acid composition is widely used in protein function analysis and classification. In the Ag-Ab interaction, some amino acid types are significantly overrepresented in epitopes, and others are underrepresented, thus the amino acid composition can be used to differentiate epitope patches from non-epitope patches [42]. For a patch, the percentage of each amino acid type is calculated as the amino acid composition.

With respect to these physicochemical and structural features, each residue in a residue patch can be represented as a feature vector of 7 dimensions (1 for relative accessible surface area, evolutionary conservation, the adjacent residue distance, and amino acid composition, respectively, 3 for secondary structure). As a result, a patch of n residues is represented by a 7 × n -dimensional feature vector.

The strategy for the imbalanced dataset

In fact, a great number of real datasets are imbalanced, in which the instances from one class take majority of the data. The common machine learning methods cannot well handle the imbalanced dataset, and they are usually combined with some strategies to solve the problem. There are two common approaches to deal with the imbalanced datasets. One approach is assigning a high cost to the misclassification of minority class and redesigning the classifier by minimizing the error rate. The other is downsizing the majority class or upsizing the minority class.

Random forest and data bootstrapping are implemented by Weka package [43], and default parameters are adopted.

Performance evaluation metrics

The performance of the models is evaluated by LOOCV and the independent test. In the study, LOOCV procedure is slightly different. For a dataset of n structures, each time, n-1 structures are used to train the model, and one structure is used to test the model. In the independent test, the prediction models are trained on the training set, and then they are tested by the independent test structures.

Where TP, TN, FP and FN are the number of true positives, the number of true negatives, the number of false positives and the number of false negatives. Here, AUC is used as the primary evaluation metric. In order to calculate AUC, we use a voting cutoff to make final prediction, and then change the cutoff to obtain different SN and SP. The scores of SN, SP, ACC and F in the following tables are calculated at the cutoff that half the number of all sub-classifiers give out the positive decision.

Performance of models based on the surface patch and thick surface patch

In Table 1, the models based on the surface patch achieve the mean AUC value of 0.611 for the patches of different sizes (from 12 to 20 residues). The models based on the thick surface patch achieve the mean AUC value of 0.619. For different patch sizes, the thick surface patch yields consistently better results than the surface patch. As shown in Table 2, the performance enhancement can be observed on the unbound dataset, regardless of the patch size. The results indicate that the thick surface patch is likely to contain more useful information for distinguishing epitope residues from non-epitope residues.

The usefulness of interior residues in the thick surface patch is further analyzed. Figure 3 shows the relative occurrence of the 20-residue thick surface patches with different number of interior residues. It is observed that most of the thick surface patches include 4, 5, 6 or 7 interior residues. The surface residues are thought to make more contribution to the epitope prediction, given the much larger number of surface residues than interior residues in the patches.

Performance of models using feature combination

According to Table 3, the relative accessible area is most important for the epitope prediction among the conventional features, with the mean AUC values of 0.570 on the bound dataset and 0.618 on the unbound dataset. ARD is a useful feature that produces the mean AUC values of 0.589 on the bound dataset and 0.627 on the unbound dataset. Moreover, the combination of conventional features and ARD yields the better results than using only conventional features, with the mean AUC scores of 0.633 and 0.654 for the bound dataset and unbound dataset, respectively.

Comparing random forest with SVM and ANN

Comparison with other methods

In recent years, several methods have been proposed to predict conformational epitopes, such as CEP [18], DiscoTope [19], PEPITO [20], ElliPro [21], SEPPA [22], Epitopia [25, 26], EPCES [27], EPSVR [28] and EPMeta [28]. According to the datasets for model training, these methods can be classified into two types, methods trained on the unbound structure and methods trained on the bound structure. Generally speaking, CEP, ElliPro, SEPPA, PEPITO, DiscoTope and Epitopia are designed to predict epitopes from bound structures, while EPCES, EPSVR and EPMeta are constructed to predict epitopes from unbound structures.

First of all, we compare our method with the bound dataset-based prediction tools on the benchmark bound dataset. According to Rubinstein's work [25], CEP, DiscoTope, ElliPro and Epitopia produce the mean AUC values of 0.53, 0.62 and 0.59 and 0.6 on the benchmark bound dataset. By using the same dataset, our method produces the mean AUC value of 0.633. The results of CEP, DiscoTope, ElliPro are obtained by their servers. We notice that part of the bound dataset has been used to build these online servers; therefore some structures may be included in both training set and test set. Obviously, the results produced by the servers of CEP, DiscoTope, and ElliPro overestimated the actual performance of these methods. The results of Epitopia and our method are produced on the same dataset by the same LOOCV procedure, and the direct comparison demonstrates the superior performance of our method. Generally, our method produces better results than these benchmark methods on the bound dataset.

Further, we compare our method with the unbound dataset-based methods on the benchmark unbound dataset. Evaluated by LOOCV, our model achieves the mean AUC value of 0.654 on the benchmark dataset. As reported in [26], EPSVR and EPCES give out the mean AUC values of 0.670 and 0.644, respectively, by using the same dataset and exactly the same assessment measures. The superior performance of our method and EPSVR is attributed to the utilization of machine learning methods. Although EPSVR gives out the better result than our method, the announced result is actually the best result that EPSVR can achieve, for EPSVR adopts the SVM parameters that give out the best result of LOOCV. Therefore, our model that adopts the default parameters produces the comparable performance.

In addition, an independent test set of 19 structures are used to compare different tools and models. The mean AUC values of DiscoTope, PEPITO, SEPPA, EPITOPIA, EPCES and EPSVR calculated by their servers are 0.567, 0.570, 0.576, 0.579, 0.586 and 0.597. Our models are trained on the benchmark bound dataset and unbound dataset, respectively, and then these models are evaluated by the independent test set. The bound dataset-based model produces the mean AUC value of 0.589 for 19 structures; while the unbound dataset-based model gives out the mean AUC value of 0.598. By trained on the same bound dataset, our model produces the better result (AUC: 0.589) than EPITOPIA (AUC: 0.579) for 19 structures. By trained on the same unbound dataset, our model produces better result than EPCES (0.598 versus 0.586), and slightly better than EPSVR. EPMeta is a meta server that ensembles the results of DiscoTope, PEPITO, SEPPA, EPITOPIA, EPCES and EPSVR, and it gives out the mean AUC value of 0.638. Nevertheless, our method is better than or comparable to any independent method in the independent test. The general tendency of the prediction precision of all methods is shown in Figure 4.

We further use the paired t-test to test differences between different methods, in which the predicted AUC scores of the test structures are used. Since the statistical analysis usually requires a great number of samples, the limited number of structures in the study leads to no statistical significance (p-value > 0.05).

It is observed that the results in the independent test are significantly poorer (AUC < 0.6) than the results in the LOOCV (AUC > 0.62 for the unbound data and AUC > 0.65 for the bound data). It is not difficult to explain the performance disparity between the independent test and LOOCV. 19 independent test structures with annotated epitope sites are collected from the CED dataset [39]. In CED, the annotated epitopes sites are actually functional epitopes determined by the wet experiment. However, computational methods for the epitope prediction focus on the structural epitopes, which are determined by the distance between antigens and antibodies (or accessible surface area loss upon antibody binding). Therefore, all methods produce the relatively poor results in independent test.

Besides structure-based prediction methods, Raghava recently proposed a method to predict conformational epitopes from antigen sequences [30]. In the method, physicochemical features (PPP), sparse encoding scheme (BPP) and amino acid composition (CCP) are used to encode the overlapping segments from antigen sequences, and prediction models are constructed by using SVM. The model based on CCP gives out the best result; therefore, it is tested and evaluated on the benchmark sequence dataset. As shown in Figure 5, all AUC values produced by the model are less than 0.6, when the window size varies from 3 residues to 15 residues. For the corresponding structure dataset, our method produces the AUC value of 0.633.

As mentioned in the introduction, PatchDock and ClusPro can be used to predict the conformational epitopes. Differing from the methods specially designed for the conformational epitope prediction, PatchDock and ClusPro focus on the protein-protein binding site prediction. Although PatchDock and ClusPro may produce the high-accuracy performance on some bound structures [38], their power for the epitope prediction is limited by some drawbacks. Firstly, we should emphasize that, due to the different prediction purposes, PatchDock and ClusPro have to use the antibodies as well as antigens while our method is identifying the potential binding sites on the antigens when antibodies are unknown. Technically, in the conformational epitope prediction, the complexes including antigens and antibodies are used to determine the binding sites on the antigens for the purpose of labelling instances. After the binding sites are labelled on the antigens, only the antigen structures can be used to train and test the prediction models because the introduction of prior knowledge about antibodies will lead to over-estimated performance. More importantly, for these docking methods, the performance on unbound structures is quite unsatisfactory in comparison to the performance on bound structures [38]. However, the unbound structure-based prediction has more practical value. Since the proposed method only requires antigen structures to make perditions and produces satisfying results on the unbound dataset, it has superiority over the docking methods in the epitope prediction.

In general, our method demonstrates overall higher prediction accuracy on the benchmark bound dataset as well as the benchmark unbound dataset.