Natural language processing in text mining for structural modeling of protein complexes

Outline of the text-mining protocol

The TM procedure was tested on 579 protein-protein complexes (bound X-ray structures purged at 30% sequence identity level) from the Dockground resource (http://dockground.compbio.ku.edu) [38]. The basic stage of the procedure consists of two major steps: information retrieval and information extraction [37] (Fig. 1). The abstracts are retrieved from PubMed using NCBI E-utilities tool (http://www.ncbi.nlm.nih. gov/books/NBK25501) requiring that either the names of both proteins (AND-query) or the name of one protein in a complex (OR-query) are present in the abstract. The text of the retrieved abstracts is then processed for the residue names. The structures of the individual proteins are used to filter the pool of the extracted residues by: (i) correspondence of the name and the number of the extracted residues to those in the Protein Data Bank (PDB) file, and (ii) presence of the extracted residue on the surface of the protein. Several NLP-based approaches (semantic similarity to generic and specialized keywords, parse tree analysis with or without SVM enhancement) were further applied for additional filtering of the extracted residues from the abstracts retrieved by the OR-queries. Performance of the TM protocol for a particular PPI, for which N residue-containing abstracts were retrieved, is evaluated as

$$ P{}_{\mathrm{TM}}\kern0.5em =\frac{\sum \limits_{i=1}^N{N}_i^{\mathrm{int}}}{\sum \limits_{i=1}^N\left({N}_i^{\mathrm{int}}+{N}_i^{\mathrm{non}}\right)}, $$

(1)

where \( {N}_i^{\mathrm{int}} \) and \( {N}_i^{\mathrm{non}} \) are the number of the interface and the non-interface residues, correspondingly, mentioned in abstract i for this PPI, not filtered out by a specific algorithm (if all residues in an abstract are purged, then this abstract is excluded from the P_TM calculations). It is convenient to compare the performance of two algorithms for residue filtering in terms of

$$ \Delta N\left(P{}_{\mathrm{TM}}\right)\kern0.5em ={N}_{tar}^{X_1}\left(P{}_{\mathrm{TM}}\right)-{N}_{tar}^{X_2}\left(P{}_{\mathrm{TM}}\right), $$

(2)

where \( {N}_{tar}^{X_1}\left(P{}_{\mathrm{TM}}\right) \) and \( {N}_{tar}^{X_2}\left(P{}_{\mathrm{TM}}\right) \) are the number of targets with P_TM value yielded by algorithms X₁ and X₂, respectively. The N(0) and N(1) values capture the general shape of the P_TM distribution. Thus, the effectiveness of an algorithm can be judged by its ability to reduce N(0) (all false positives) and increase N(1) (all true positives). In this study, advanced residue filtering algorithms are applied to the pool of residues extracted by the OR-queries with the basic residue filtering, thus X₂ will hereafter refer to this algorithm. The negative values of ΔN(0) and the positive values of ΔN(1) indicate successful purging of irrelevant residues from the mined abstracts.

Selection of keywords

Generic keywords semantically closest to PPI-specific concept keywords (see Results) were found using Perl module QueryData.pm. The other Perl modules lesk.pm, lin.pm and path.pm were used to calculate similarity scores introduced by Lesk [39, 40], Lin [41] and Path [42, 43], correspondingly, between the token (words) in a residue-containing sentence and the generic keywords. These Perl modules, provided by the WordNet [44, 45] (http://wordnet.princeton.edu), were downloaded from http://search.cpan.org. The score thresholds for the residue filtering were set as 20, 0.2, and 0.11, for the Lesk, Lin and Path scores, respectively.

The keywords relevant to the PPI binding site (PPI + ive words), and the keywords that may represent the fact of interaction only (PPI-ive words) (Table 3) were selected from manual analysis of the parse trees for 500 sentences from 208 abstracts on studies of 32 protein complexes.

Scoring of residue-containing and context sentences

An add-on value to the main S_X score (Eq. 3) from the context sentences (sentences immediately preceding and following the residue-containing sentence) was calculated either as simple presence or absence of keywords in these sentences, or as a score, similar to the S_X score, but between the keywords and the root of the sentence on the parse tree.

SVM model

Text mining constraints in docking protocol

The residues at the co-crystallized interface were used as reference. Such residues were determined by 6 Å atom-atom distance across the interface. The reference residue pairs were ranked according to the C^α - C^α distance. The top three residue-residue pairs were used in docking with the highest confidence score 10, to determine the maximum possible success rate for the protein set.

Generic and specialized dictionaries

The simplest approach to examining the context of a residue mentioned in the abstracts would be to access the semantic similarity of words (token) in the residue-containing sentence to a generic but at the same time PPI-relevant concept. For the purpose of this study, such concept was chosen to be “binding site” as the one describing the physical contact between the two entities (proteins). We designated the words “touch” and “site” as the most semantically similar words relevant to this concept (binding site) to be used in WordNet [44, 45] (generic English lexical database with words grouped into sets of cognitive synonyms), which does not contain any knowledge-domain specific vocabularies [53]. Thus, we calculated similarity scores (see Methods) between these two words and all the words of the residue-containing sentence(s) in the abstracts retrieved by the OR-query. If a score exceeded a certain threshold, all residues in the sentence were considered to be the interface ones. Otherwise they were removed from the pool of the mined residues. The calculations were performed using three different algorithms for the similarity score. Similarity scores by Lesk and Path demonstrated only marginal improvement in the filtering of mined residues compared to the basic residue filtering (Table 1 and Fig. 2). Lin’s score yielded considerably worse performance. Similarly poor performance of this score was reported previously, when it was applied to word prediction for nouns, verbs and across parts of speech [54]. In our opinion, this may be due to some degree of arbitrariness in the way the similar words are grouped under a common subsumer (most specific ancestor node), and how this subsumer fits into the overall hierarchy within the synset (set of cognitive synonyms). Thus, we concluded that generic vocabularies cannot be employed in the TM protocols for identifying PPI binding sites. This correlates with the conclusions of Sanchez et al. [55] that hierarchical structure of generic and domain-specific vocabularies are different and thus, for example, MESH specific vocabulary [56] provides more accurate knowledge representation of medical concepts compared to the generic WordNet lexicon.

Query	Similarity measure	L tot a	L int b	Coverage (%)c	Success (%)d	Accuracy (%)e	ΔN(0)f	ΔN(1)f
AND	–	128	108	22.1	18.7	84.4
OR	–	328	273	56.6	47.2	83.2
OR	Lesk [39, 40]	319	267	55.1	46.1	83.7	-3	−1
OR	Lin [41]	251	184	43.4	31.8	73.3	+ 8	−8
OR	Path [42, 43]	316	265	54.6	45.8	83.9	−3	+ 1

^aNumber of complexes for which TM protocol found at least one abstract with residues

^bNumber of complexes with at least one interface residue found in abstracts

^cRatio of L _tot and total number of complexes

^dRatio of L _int and total number of complexes

^eRatio of L _int and L _tot

^fCalculated by Eq. (2)

Next, we tested applicability of the 7 specialized dictionaries (Table 2) to filtering of the residues mined by the OR-queries. All these dictionaries were specifically designed for the mining of the literature on PPI identification and contain up to several hundred PPI-relevant keywords. Thus, there is no need to measure semantic similarity between words in the residue-containing sentence and words in these dictionaries, and it is just enough to spot these words in the sentences (maximum possible semantic similarity). If any keyword was spotted in a sentence, all residues mentioned in this sentence were considered as interface residues. The results (Table 2 and Fig. 3) indicated, however, that using all dictionaries did not yield significant improvement in the residue filtering. While some dictionaries (with ΔN(0) < 0 in Table 2) succeeded in removing irrelevant information, there is a general tendency of removing relevant information as well (predominantly negative numbers of ΔN(1) in Table 2). Interestingly, the best performing dictionary by Schuhmann et al. [57] contains the smallest number of words.

Dictionary and reference	Number of PPI keywords	L tot a	L int b	Coverage (%)c	Success (%)d	Accuracy (%)e	ΔN(0)f	ΔN(1)f
Blaschke et al., [20]	43	265	205	45.8	35.4	77.4	0	−8
Chowdhary et al., [58]	191	284	233	49.1	40.2	82.0	−7	−4
Hakenberg et al. [59]	234	297	232	51.3	40.1	78.1	6	−7
Plake et al. [60]	73	291	230	50.3	39.7	79.0	1	−1
Raja et al. [23]	412	302	247	52.2	42.7	81.8	0	−5
Schuhmann et al. [57]	64	212	152	36.6	26.3	71.7	− 1	5
Temkin et al. [21]	174	283	223	48.9	38.5	78.8	0	−9
Own dictionary	16	224	169	38.7	29.2	75.4	−6	8

For definitions of columns 3–9, see footnotes to Table 1. Full content of in-house dictionary is in Table 3, but only PPI + ive part was used to calculate the data in this Table

All tested dictionaries were designed for the mining information on the existence of interaction. Thus, we also tested our own dictionary, designed specifically to distinguish keywords relevant and irrelevant to the protein-protein binding sites (see Methods). Despite the small amount of PPI-relevant words in the dictionary, the filtering of the mined residues based on this dictionary led to considerable improvement in the TM performance (the rightmost bars in Fig. 3 and the bottom row in Table 2). This suggests that even a limited amount of text provided by abstracts can be used to extract reliable PPI-relevant keywords.

Analysis of sentence parse tree - deep parsing

The interface residues tend to have S_X > 0.25 (Additional file 1: Figure S8). Thus, we used this value as a threshold to distinguish between interface and non-interface residues. Compared to the simple dictionary look-up (see above), even such simplified analysis of the parse tree, yielded significant improvement in the performance of our text-mining protocol (Method 1 in Table 4 and red bars in Fig. 4).

Method of parse tree analysis	L tot	L int	Coverage (%)	Success (%)	Accuracy (%)	ΔN(0)	ΔN(1)
Method 1. Scoring of the residue-containing sentence only	222	173	38.3	29.9	77.9	−13	+ 10
Method 2. Scoring of the residue-containing sentence and keyword spotting in the context sentences	208	154	35.9	26.6	74.0	−7	+ 3
Method 3. SVM model with scores of the residue-containing and context sentences	182	146	31.4	25.2	80.2	−27	+ 21

Keywords used in the analysis were taken from our dictionary (Table 3). For definitions of columns 2–8, see footnotes to Table 1

The main message of a sentence can propagate through the article text comprising several sentences around the master sentence (context) and therefore it would be logical to include context information in the residue filtering as well. However, there is no clear understanding how far away the message can spread, especially in such dense text as an abstract. Thus, we treated as context only sentences immediately preceding and following the residue-containing sentence. These sentences usually do not contain residues. Thus, we included context information either by simple spotting PPI + ive keywords in these sentences (Method 2) or by calculating S_X-like score of PPI + ive and PPI-ive words with respect to the sentence root (Method 3). In the former algorithm, a mined residue is treated as interface residues if its S_X > 0.25 and a PPI + ive keyword was spotted in the context sentences. The latter algorithm requires a more complicated approach as there is no clear distinction between the context-sentence scores for interface and non-interface residues. Thus, classification of the residues was performed by an SVM model with the optimal parameters (see Methods).

Inclusion of the context information by simple keyword spotting worsens the performance of the residue filtering (Method 2 in Table 4 and cyan bars in Fig. 4) as many interface residues are erroneously classified due to the absence of the keywords in the context sentences. Application of the SVM model, despite a relatively small number of its features, increased filtering performance dramatically, making SVM-based approach superior to all other methods investigated in this study. All three methods have comparable values of overall success and accuracy (Table 4). An example of successful filtering of non-interface residues is shown in Fig. 5 for the chains A and B of 2uyz. Out of five residues mined by the basic TM protocol, only one residue (Fig. 5, Glu67B) was at the complex interface (P_TM = 0.20). SVM model has filtered out all four non-interface residues, elevating TM performance to P_TM = 1.00 (details are available in Additional file 1: Table S1 and accompanying text).

Finally, to ensure that the results are not determined by over fitting the SVM model, we filtered residues on a reduced set of abstracts where all abstracts for a complex were excluded from the consideration if at least one abstract contained sentence(s) used for the training of the SVM model. Despite a significant drop in the coverage, the results on the reduced set (Additional file 1: Figure S9) did not differ much from the results obtained on the full set of abstracts.

Docking using text-mining constraints

Constraints generated by NLP were tested in docking by GRAMM to model complexes of unbound proteins from the Dockground X-ray benchmark set 4 (see Methods). The set consists of 395 pairs of separately resolved unbound protein structures and their co-crystallized complexes. Each unbound complex was docked by GRAMM three times, using (1) constraints from the basic TM, (2) constraints re-ranked by NLP, and (3) the reference constraints. The output of the global low-resolution docking scan consisted of 20,000 matches, with no post-processing (except for the removal of redundant matches). The matches were scored by the sum of the f values (Eq. 7), if constraints were generated for the complex. If no constraints were generated, the score was zero. The quality of a match was assessed by C^α ligand interface root-mean-square deviation, i-RMSD (ligand and receptor are the smaller and the larger proteins in the complex, respectively), calculated between the interface of the docked unbound ligand and the corresponding atoms of the unbound ligand superimposed on the bound ligand in the co-crystallized complex. Success was defined as at least one model with i-RMSD ≤5 Å in top 10 predictions. The results (Fig. 6) show significant success rate increase in the docking output when using constraints generated by the advanced TM, from 27% in the case of the basic TM, to 47% in the case of the advanced TM with NLP.

Since some authors might not include the required details in the abstracts of their papers, we plan to extend the automated analysis to the full-text articles, as well as to explore incorporation of the papers from bioaRxiv. This should increase of the size of the training sets for machine-learning models, and the number of available features, thus enabling the use of the deep learning methodologies for generation of the docking constraints. Such constraints could be potentially further improved by incorporating information automatically extracted from other publicly available PPI-related resources, leading to more accurate and reliable structural modeling of protein interactions.