Workflow and auxiliary means

The DRENDA work flow is based on the disease mining procedures already included in BRENDA [13] but strongly enhanced inter alia by the added classification feature. The work flow (Figure 1) is implemented in Python and uses MySQL as database back-end. The first step is the initial sentence splitting (Figure 1, Sentence splitting) and search for the co-occurrence of enzyme and disease terms (Figure 1, Co-occurrence matching) within the title or an abstract sentence of the references enlisted in PubMed. The BRENDA enzyme names and synonyms, derived from manual literature annotation, comprise currently around 100,000 terms. This collection is certainly the largest available reference dictionary for the identification of enzyme entities (Figure 1, BRENDA).

The terms for disease recognition are taken from the Me dical S ubject H eadings (MeSH) [14]. The selected disease terms are a subset containing about 22,000 disease names and synonyms (Figure 1, MeSH). The titles and sentences, where at least one co-occurring enzyme and disease entity could be found (Figure 1, Result set co-occurrence) are the basis of the DRENDA content and stored as corpus for the subsequent classification.

Relation classification

The classification is accomplished by machine learning methods. The basic classification unit is a sentence or title from PubMed abstracts where at least one enzyme and disease entity could be found (Figure 1, Result set co-occurrence). As they contain only a small number of words the problem is characterised by a sparse feature representation and offers the possibility to perform the classification via a S upport V ector M achine (SVM) that have proven to perform well on text classification tasks [15]. We have used SVMlight [16] as SVM implementation. In order to use an SVM for classification the availability of training and testing material is an essential requirement. For this purpose a corpus has been composed (Figure 1, Test corpus; Train corpus), which contains about 5,031 sentences and titles derived from PubMed abstracts. The selection out of all sentences and titles was made accordingly the following conditions: 2,500 have been randomly selected where at least one enzyme entity was identified. 2,000 have been randomly selected where at least one enzyme entity and one disease entity was identified. About 500 were randomly collected without any precondition from PubMed abstracts and titles. This corpus was manually annotated by an expert with biomedical background, for the identification of the presence of enzyme and disease entities and the assignment of all categories. If both entities are present within one sentence or title the relation was manually classified into the four categories of causal interaction, therapeutic application, diagnostic usage and ongoing research. In addition to the deployment as training and test data it is also used to evaluate the co-occurrence matches (Figure 1, Evaluation).

Classification categories

Preprocessing and representation

In order to form a suitable input for SVM classification the set of titles and sentences, which were classified to contain a co-occurrence of an enzyme and disease, are prepared in a preprocessing step (Figure 1, Preprocessing).

First, terms that represent the entities "enzyme" and "disease" are removed (removal) or replaced (replacement) by a general term representing all enzyme respectively disease terms. This means that in the removal method all terms are deleted which are known to represent names and synonyms of enzymes or diseases without any substitution. In the replacement method the deleted names are replaced by a single generic term for all diseases and one generic term for all enzymes. Given that these two generic terms not have been present in the sentences and titles of the result set co-occurrence before their insertion during the preprocessing process.

The titles and sentences are represented as feature vectors within a document space [22]. The feature vector representation of a sentence or a title is performed via a calculation of the term frequency inverse document frequency tf-idf. The term frequency tf _ij is the number of occurrences of the term t _i in the particular sentences or title d _j . The inverse document frequency idf _i is calculated as

(1)

where D is the overall number of sentences and titles and d _it is the number of sentences and titles containing t _i . tf-idf is the product of tf _ij and idf _i .

(2)

Finally, all coordinates are divided by the length of the feature vector. Thereby each feature vector is converted to unit vector length, neutralizing the influence of the different sentence lengths.

Classification of the co-occurrence results

As the next step of the described DRENDA work flow a classification of the co-occurrence results is performed. The test/training corpus has been randomly split into 4/5 training examples and 1/5 test examples. Every discrete classification category contains the same quantity of positive and negative annotated sentences/titles. The overall number of examples differs between each classification category due to the amount of positive assigned sentences/titles in each category. All sentences/titles which were positively annotated for one category are joined with the same amount of sentences/titles, negatively annotated for this category and containing at least one co-occurrence of an enzyme and disease entity.

A classification model (Figure 1, SVM training) is calculated and used to perform the classification of co-occurrence results (Figure 1, SVM classification). The classification results are finally distributed into a four level confidence system with descending classification precision and specificity from level 4 to 1.

Cross validation

Before establishing the described DRENDA work flow (Figure 1) a five-fold cross-validation was performed to test the performance of the preprocessing steps and methods (removal; replacement) and the appropriate parameter choice for SVMlight. The overall ability of classification and assignment of sentences to the predefined categories was validated. In every category, causal interaction, therapeutic application, diagnostic usage and ongoing research, the same quantity of positive and negative annotated sentences/titles was chosen out of the training/test corpus and split randomly into five sets of equal size. Each individual set served once as a testing set while the other four formed a training set. Several calibrations of SVMlight parameters were tested as well as all four default kernel functions implemented in SVMlight (linear, polynomial, radial basis, sigmoid) were tested. This led to 2,688 distinct parameter combinations processed for each preprocessing method, removal and replacement. The classification of the co-occurrence results in the DRENDA work flow is processed once and not according to the cross validation scheme, due to processing time.

Evaluation measures

In order to evaluate the correctness of the results derived from the co-occurrence based entity recognition (Figure 1, Result set co-occurrence) and the entity relation classification (Figure 1, Classified sentences and titles) steps in the DRENDA work flow some commonly used measures, including precision, recall, accuracy, specificity, F1 score and Matthews correlation coefficient (MCC), were calculated based on the test sets formed from the manually annotated corpus (Figure 1, Test corpus). The correct estimated presence of an enzyme and disease entity in the co-occurrence is counted as a true positive (tp), the correct estimated absence of one or both is counted as true negative (tn). If there are no co-occurring enzyme or disease entities or even one entity is missing but erroneously denoted as co-occurring it is considered as a false positive (fp) prospect. The case of co-occurring entities present which are not found by the routine is considered as false negative (fn). In the classification the correct assignment of a sentence to one of the classification categories is assessed as tp. The incorrect assignment to a classification category is assessed as fp. The correct neglected assignment to a classification category is assessed as tn and the incorrect neglected assignment as fn. The evaluation measures are defined as:

(3)

(4)

(5)

(6)

(7)

(8)

(9)

In the five-fold cross-validation the values are also accompanied by the standard deviation.

In order to evaluate the benefit of a threshold variation to the predictive performance of the classification models receiver operating characteristic (ROC) curves have been plotted for the results of the five-fold cross-validation. In these ROC curves the average recall, or true positive rate, was plotted versus the false positive rate. The ROC plots and the calculation of the area under the curve (AUC) were performed by the use of the R package ROCR [23].

Enzyme and Disease related references

The content of DRENDA focuses on information on enzymes and ignores other proteins which may also be of biomedical relevance in a physiological and pathophysiological context. This selection is caused by the available extended dictionaries of enzyme names and on the close relationship of DRENDA as a text mining derived knowledge source which contributes to the expert annotated information of BRENDA [7].

Notably peptidases (EC sub-class 3.4.) catalysing the cleavage of peptides and proteins form 53% of the DRENDA entries of hydrolases. They are the EC sub-class with the highest quantity of distinct EC numbers, containing with 41% a remarkable part of all hydrolases classified so far. Peptidases perform many essential functions within eukaryotic life e.g. in the blood coagulation cascade (thrombin, plasmin), the immunologic complement system (complement protease C1r, complement factor D), digestion of food proteins (pepsin, trypsin). Thus a deficiency of a peptidase may lead to manifold pathological processes. Their full functionality e.g. as virulence factors in bacteria (anthrax lethal factor endopeptidase) is also of pathogenic importance. In BRENDA about 20% of the inhibitor-related entries are connected to peptidases. This increased emergence indicates a high research interest especially in the medical field. Inhibition of a proteolytic enzyme is often a part of a potential therapeutic strategy. In comparison, other enzyme classes are not as strongly represented. The ligases (EC class 6) are rather infrequently connected to diseases (table 1). In BRENDA only 3% of the inhibitor-related entries are connected to ligases.

The results of the co-occurrence based disease and enzyme entity recognition were compared to the corresponding annotations of the test/train corpus. The co-occurrence related F1 score is up to 88.8% and shows an accuracy of 89.8%. The large collection of enzyme names in the reference dictionary is both an advantage and a disadvantage. Homonyms are one form of ambiguity. These are words that share the same spelling but have different meanings. They may lead to false positive matches. This originates from the use of general terms for enzymes, like the use of "lethal toxin" for anthrax lethal factor endopeptidase (EC 3.4.24.83), an enzyme that catalyses the hydrolysis of mitogen-activated protein kinase kinases and is secreted by virulent strains of the bacterium Bacillus anthracis as an enzymatic part of its exotoxin. Short acronyms in particular tend to be homonyms like "AAA", one of the synonyms for aryl-acylamidase (EC 3.5.1.13). "AAA" can have diverse meanings in a (bio-) medical article e.g. abdominal aortic aneurysm or American Ambulance Association. Where possible the dictionaries are optimized with respect to a deletion of homonyms and any ambiguous terms. The deletion of terms has to be considered with care because ignoring this terms also leads to a certain amount of missed entities (false negative) and thus the recall value decreases. Moreover there are other pitfalls in bio-entity recognition. There is a tendency in the literature to create new terms for enzyme disregarding the established systematic nomenclature. This increases the effort to keep dictionaries up-to-date with all used synonyms. Spelling mistakes also increase the difficulty in identifying entities.

Classification performance check

The classification feature included in DRENDA was tested in a five-fold cross-validation. The F1 score for classification ranges between 0.802 ± 0.032 and 0.738 ± 0.033 (0.583 ± 0.076 and 0.395 ± 0.058 corresponding MCC) depending on the category and pre-processing procedures (table 2). One of the eminent observations of the cross-validation results is the rather small influence of the preprocessing methods of removal (table 2, a) and replacement (table 2, b) of terms representing the entities "enzyme" and "disease" on the F1 score and the AUC values (table 2) of the ROC plots in Figure 2.

Category	Preprocessing mode	F1 score	MCC	AUC
therapeutic application	a	0.802 ± 0.032	0.583 ± 0.076	0.878 ± 0.040
ongoing research	a	0.733 ± 0.024	0.395 ± 0.058	0.752 ± 0.037
diagnostic usage	a	0.738 ± 0.032	0.430 ± 0.074	0.784 ± 0.032
causal interaction	a	0.743 ± 0.009	0.429 ± 0.023	0.788 ± 0.020
therapeutic application	b	0.792 ± 0.016	0.548 ± 0.041	0.878 ± 0.038
ongoing research	b	0.744 ± 0.020	0.427 ± 0.051	0.752 ± 0.037
diagnostic usage	b	0.732 ± 0.022	0.412 ± 0.075	0.783 ± 0.033
causal interaction	b	0.742 ± 0.010	0.428 ± 0.024	0.786 ± 0.020

The category with the best F1 score is therapeutic application with 0.802 ± 0.032 for the removal preprocessing and 0.792 ± 0.016 for replacement preprocessing. This category shows also the maximum AUC values (0.878 ± 0.040 removal, 0.878 ± 0.038 replacement) and the maximum MCCs (0.583 ± 0.076 removal, 0.548 ± 0.041 replacement). The F1 scores of the categories causal interaction and ongoing research are comparable.

However, with an F1 score of 0.744 ± 0.020 and an MCC of 0.427 ± 0.051 for the replacement preprocessing, this method is more successful for the category ongoing research than the removal preprocessing (F1 score of 0.733 ± 0.024, and an MCC of 0.395 ± 0.058. In all other categories the removal of the terms representing the entities "enzyme" and "disease" achieved slightly better or equal results compared to the replacement.

In the five-fold cross-validation two preprocessing methods and 2,688 distinct parameter combinations were evaluated. Results for the models with the maximal achieved F1 scores, their corresponding Matthews Correlation Coefficient (MCC) and the area under the curve (AUC) of their receiver operating characteristic (ROC) plots shown in Figure 2. In the removal method (a) all terms representing the entities "enzyme" and "disease" were removed before the calculation of term weights, in the replacement method (b) these terms were replaced by a general term standing for all enzyme respectively disease terms.

A stop word removal with different combinations of high-frequency stop words was tested, but lowered the quality of the classification results (data not shown) so that it was decided not to remove stop words at all, apart from the described removal of terms representing enzyme and disease entities.

A word stemming stage was not included. This saves calculation time especially for processing a large corpus which is the result of the search for co-occurring enzyme and disease entities. Despite the omission of a word stemming processing step we achieve acceptable results regarding quality and quantity and could not observe any fatal influence on classification results.

Interrelations of enzyme disease combinations

All sentences where an enzyme and a disease entity had been identified by the co-occurrence matching are classified by an SVM. The text corpus, consisting of the sentences and titles of the co-occurrence results (Figure 1, Result set co-occurrence), was preprocessed according to both methods removal and replacement of entity terms. For the classification of the co-occurrence corpus a reduced variation of the 57 best parameters of the 5-fold cross validation have been chosen. That comprises all parameter sets where a precision of at least 0.7 and an F1 score of at least 0.5 in one or more classification category. Each reference is assigned to one or more of the predefined enzyme and disease relation categories causal interaction, therapeutic application, diagnostic usage and ongoing research. The classification decision is accomplished by a binary SVM classification of each sentence or title. A reference can be assigned to none, one or more categories.

Table 3 shows that 200,137 of the 910,897 combinations are assigned to the category causal interaction with a specificity of 96.7% in confidence level 4.

For each confidence level the number of matches in the category causal interaction is higher than in the other three categories (table 3, table 4, table 5 and table 6). A detailed view on the overlap of the four categories is given in Figure 3. A large subset of the categories therapeutic application, diagnostic usage and ongoing research overlaps with the set causal interaction. Throughout all four confidence levels about 50% of the results assigned to the particular category therapeutic application, diagnostic usage or ongoing research (Figure 3b-d) are also members of the category causal interaction (Figure 3).

This, of course, is expected as the fact that an enzyme causes a disease implies, that it often can be used in diagnosis or for therapy. For example the Gaucher's disease, a lysosomal storage disease which is caused by a hereditary deficiency of the enzyme glucosylceramidase (EC 3.2.1.45), is treated by a lifelong enzyme replacement therapy [24] and thus the enzyme is also the therapeutic agent. However, the proportion of entries belonging to all four categories decreases strongly with increasing confidence level. The selection in the higher confidence levels tends to prefer unequivocal statements and more ambiguous statements are rejected which reduces the number of entries intersecting all categories.

Prostaglandin-endoperoxide synthase and its multiple relevance

Prostaglandin-endoperoxide synthase (EC 1.14.99.1), one of the most frequently retrieved enzymes, is registered with 15,103 entries associated to 832 diseases. It catalyses the initial step in the formation of prostaglandins, being the central enzyme in the biosynthetic pathway from arachidonic acid to prostaglandins.

The tissue distribution and physiological functions are depending on the isoform of prostaglandin-endoperoxide synthase. It is involved in platelet aggregation, gastric mucosa protection and renal electrolyte homeostasis [25], the inducible isoenzyme of prostaglandin-endoperoxide synthase is involved in inflammation processes and mitogenesis. It is expressed in response to stimuli like cytokines, growth factors, or hormones. Furthermore, the overexpression of the inducible isoenzyme is observed in pre-malignant and malignant neoplasms of the colon, liver, pancreas, breast, lung, bladder, skin, stomach, head and neck [26]. Prostaglandin-endoperoxidase synthase is a target of nonsteroidal anti-inflammatory drugs like Ibuprofen and Acetylsalicylic acid and is relevant in the treatment of mild to moderate pain, fever, inflammation processes and cardiovascular and rheumatic diseases. Hence it is not surprising that in each DRENDA confidence level it is one of the enzymes which ranges throughout in the top ten in the intersection set of all four classification categories.

Part of the training/test corpus was manually reannotated by a second expert with a background in biochemistry and enzymology in order to determine the inter-curator agreement. Reannotation was performed without knowledge of the annotation of the first curator (biomedical background). The amount of the inter-curator agreement ranged between 89.6% (therapeutic application) and 66.0% (ongoing research) with most of the differences occurring in the between ongoing research classification and one of the others. In many cases the difference occurred in the assessment whether statement made by the authors is strong enough to put it into one of the defined categories causal interactions, diagnostic or therapeutic or whether it is additional research is needed.

Comparison with DrugBank

Accession to DRENDA

The DRENDA disease and enzyme related data are available via the BRENDA web portal http://www.brenda-enzymes.org in the section "Disease/Diagnostics" (Figure 4). The DRENDA data are fully searchable via a query form (Figure 5a). The fields provided allow searching a full or partial name of a disease, an enzyme, an EC number or the title of a publication. These fields can individually be combined for an optimal refinement of the query. By the selection of the check boxes of the fields "Category" or "Confidence Level" the classification results of the retrieved references are visible. By submitting the query a result table is created, according to the requested information (Figure 5b). Every entry contains links to the BRENDA flat file view of the enzyme, a search for amino acid sequences of the enzyme, a view on the catalysed reaction and a link to the abstract of the reference to PubMed. Thus DRENDA results are effectively connected to gain close information about the enzyme from the BRENDA information system.