Comparing a knowledge-driven approach to a supervised machine learning approach in large-scale extraction of drug-side effect relationships from free-text biomedical literature

It has been increasingly recognized that similar side effects of seemingly unrelated drugs can be caused by their common off-targets and that drugs with similar side effects are likely to share molecular targets [1]. Therefore, systems approaches to studying side effect relationships among drugs and integration of this drug phenotypic data with drug-related genetic, genomic, proteomic, and chemical data will facilitate drug target discovery and drug repositioning. The availability of a comprehensive drug-side effect (SE) relationship knowledge base is critical for these tasks. Current drug phenotype-driven systems approaches rely exclusively on drug-SE associations extracted from FDA drug labels. However, there exists a large amount of additional drug-SE relationship knowledge in the large body of published biomedical literature. In this study, we present a novel knowledge-driven approach to automatically extract a large number of drug-SE pairs from 21 million published biomedical abstracts. We systematically analyzed extracted drug-SE pairs in combination with drug-related gene targets, metabolism, pathways, gene expression and chemical structure data. We show that these extracted drug-SE pairs have great potential in drug discovery.

Systems approaches to studying the phenotypic relationships among drugs can facilitate rapid drug target discovery and drug repositioning. Computational approaches to predicting drug targets have often been based on chemical similarity measures and docking strategies [7, 16]. Similarly, many computational strategies for drug repositioning have been explored [6]. The majority of these approaches leverage on known drug properties such as chemical similarity [7], molecular activity similarity [12], molecular docking [8], and gene expression profile similarity [13]. In a seminal paper, Campillos et al. used phenotypic side-effect similarities among drugs to predict new targets for drugs [1]. However, their analysis was limited to drug-SE relationships derived solely from the FDA drug labels. In one of our recent studies, we show that much of the drug-SE association knowledge from biomedical literature has not been captured in FDA drug labels yet [17].

Currently, more than 21 million biomedical records are available on MEDLINE. While many biomedical relationship extraction tasks have focused on extracting relationships between drugs, diseases, proteins, or genes [2, 18, 19], extracting drug-SE relationships from MEDLINE has been less explored. Recently, Gurulingappa et al. trained and tested a supervised machine learning classifier to classify drug-condition pairs in a set of 2972 manually annotated case reports [4]. That study focused on a limited set of drugs and side effects and case reports. It is unclear how their approach can be effectively scaled up to the whole MEDLINE in building a large-scale drug-SE relationship knowledge base. Recently, we developed an approach in boosting drug safety signal detection from FDA Adverse Event Reporting System (FAERS) using evidence from MEDLINE [20]. We developed an automatic approach to extract anticancer drug-specific side effects from MEDLINE by developing specific filtering and ranking schemes [21]. We developed a pattern-based learning approach to accurately extract drug-SE pairs from MEDLINE sentences [22]. We combined automatic table classification and relationship extraction in extracting anticancer drug-side effect pairs from full-text articles [23]. In this study, we present a knowledge-driven (KD) text-classification-based approach to extract drug-SE pairs from MEDLINE sentences. Different from our previous studies where we extracted drug-SE pairs from unclassified sentence, here we classified sentences into drug-SE-related and -unrelated before relationship extraction. Our approach is also different from other text classification-based approaches that often trained text classifiers using annotated training datasets to find drug-SE-related sentences [4], instead, we implicitly classified MEDLINE sentences using known drug-SE pairs. Since our study did not explicitly train a text classifier, it is highly dynamic and effective: it can easily incorporate any changing prior knowledge and quickly extracted drug-SE pairs from the whole MEDLINE (21,354,075 abstracts and 119,085,682 sentences).

Our study is based on the two key observations: (1) multiple side effects for a drug are often reported in the same sentences or abstracts; and (2) if a sentence contains a known drug-SE pair, then this sentence is likely to be SE-relevant. Other pairs in this SE-related sentence are likely to be drug-SE pairs. For example, the sentence "At the final irinotecan dose of 50 mg/m(2), grade 3 or higher toxicity included diarrhea (26%), neutropenia (21%), nausea (18%), fatigue (16%), anorexia (13%), and thrombosis/embolism (13%)" (PMID 19139178) contains a known drug-SE pair "irinotecan-diarrhea." Based on this fact, we know that this sentence is SE-related and that the other five pairs in this sentence are likely to be drug-SE pairs. On the other hand, the following sentence "Weekly docetaxel, cisplatin, and irinotecan (TPC): results of a multicenter phase II trial in patients with metastatic esophagogastric cancer " contain no known drug-SE pair, therefore no pair will be extracted from this sentences even though it contains three drug-disease pairs. In this study, we used all known drug-SE pairs derived from FDA drug labels as prior knowledge to find SE-related MEDLINE sentences and abstracts, from which many additional drug-SE pairs that have not included in FDA drug labels are then extracted. We compared the KD approach to a support vector machine (SVM)-based approach.

The entire experimental process consists of the following steps: (1) build a local MEDLINE search engine; (2) develop, evaluate and compare the KD approach to a SVM-based approach; (3) extract drug-SE pairs from MEDLINE; and (4) systematically analyze the correlation between drug-associated side effects and drug gene targets, metabolism genes, chemical similarity, and disease indications.

Develop, evaluate and compare the KD approach to the SVM-based approach

We downloaded a total of 100,049 known drug-SE pairs from SIDER (Side Effect Resource), a public, machine-readable side effect resource that was automatically constructed from FDA package inserts [10]. These pairs are used as prior knowledge for the KD approach. Using each drug-SE pair from the prior knowledge data as a search query to the local MEDLINE search engine, we retrieved all MEDLINE sentences and abstracts that contain at least one known drug-SE pair. These sentences are determined as SE-related. We then extracted drug-SE co-occurrence pairs from these SE-relevant sentences, with the restriction that both drug and SE names must be noun phrases in the parse trees of the sentences. We have recently shown that this restriction can increase the precision of biomedical relationship extraction from MEDLINE [19–22]. The drug and SE lists (996 drugs (generic names) and 4,199 SE terms) are from SIDER.

We compared the drug-SE extraction from sentences classified using the KD approach with that from sentences classified using a SVM-based text classifier (described later). For comparison, we selected ten drugs from SIDER that are associated with the most numbers of SEs and compared the performance of KD approach to the SVM approach in extracting drug-SE pairs for each of them (Figure 1). For each drug, we randomly split its drug-SE pairs into two equal parts: training dataset and testing dataset. For the KD approach, we first retrieved all sentences that contain at least one of the 10 drugs and at least one SE term from the SE lexicon (4,199 SE terms). We then classified these sentences into SE-related and -unrelated. A sentence is determined as SE-related if it contains at least one known drug-SE pair from the training dataset (prior knowledge). We then extracted additional drug-SE pairs from these sentences.

For the SVM-based approach, we first classified MEDLINE sentences into drug-SE-related and -unrelated using a pre-trained SVM-sentence classifier. We then extracted drug-SE co-occurrence pairs from positively classified sentences. A two-class SVM-based sentence classifier was trained using implementation in WeKa [5]. The positive training data consisted of a total of 320,175 sentences, each of which contained at least one known drug-SE pairs from SIDER (the testing data for above 10 drugs were excluded). Equal number of negative sentences was randomly selected from the rest of MEDLINE sentences. The SVM-based sentence classifier used polynomial kernel, bag-of-words feature, TF-IDF weighting, stemming and stopwords-removal. The bag-of-words feature was used since it is often the case that the appearance of one specific word such as 'toxicity' can be used to determine whether a sentence is drug-SE-related. The 10-fold cross validation was used in training the SVM classifier. For both KD and SVM-based approaches, the input sentences are the same, which are sentences that contain at least one of the 10 drugs and at least one term from the SE lexicon. We evaluated and compared the performance using the same testing datasets. Precisions, recalls and F1 scores for these 10 drugs were calculated.

Our current study has several limitations and can be significantly improved in future studies. First, we used drug-SE pairs from SIDER as prior knowledge for the KD approach. The overall performance depends on the accuracy and comprehensiveness of the SIDER database. Errors and uncertainties in the knowledge base can propagate into the relationship extraction process and adversely affect the precision. For example, the drug-disease treatment pair 'ondansetron-pain' was incorrectly specified as a drug-SE pair in SIDER. Because of this error, our algorithm classified the following sentence as SE-related: "Ondansetron, lidocaine, tramadol, and fentanyl were effective in preventing and decreasing the level of rocuronium injection pain" (PMID 12032018). Three additional pairs (lidocaine-pain, tramadol-pain, and fentanyl-pain) were incorrectly extracted as drug-SE pairs. Since SIDER was constructed from FDA drug labels using text-mining approaches, errors may be inevitable for completely automatic method. Currently, we are manually extracting drug-associated side effects from FDA drug labels. Second, our algorithm cannot extract correct pairs from sentences with multiple drugs and multiple side effect names (n × m), even though the sentences are side effect-related. For example, sentence "... decreases in hemoglobin, nausea/vomiting, and hyperbilirubinemia were observed to be influenced by the previous use of irinotecan (OR = 3.07, P = 0.003), mitomycin (OR = 2.28, P = 0.004), and cisplatin (OR = 1.60, P = 0.007), respectively" (PMID: 17577624). Three drugs and three SEs are specified in the sentence, but only three, instead of 9 (3 × 3) are valid drug-SE pairs. This is a difficult problem for not only the KD approach but also for automatic relationship extraction in general. In this case, human curation may be necessary. Even with the above mentioned limitations, we demonstrated that the large number of drug-SE pairs extracted from MEDLINE reflect drug-related genetic, genomic and chemical information and can have potential in computational drug target discovery and drug repositioning. Currently, we are developing integrative systems approaches for drug repositioning by fully exploiting data ranging from lower level genetic connections to immediate layer genomic data to higher level phenotype data in order to build integrative models of genetic, genomic, and phenotypic complexity.

We have developed a novel KD approach in extracting a total of 49,575 drug-SE pairs from 119,085,682 MELDINE sentences and 180,454 pairs from 21,354,075 MEDLINE abstracts (records). We show that the KD approach performed significantly better that a SVM-based machine approach. We demonstrated that this large-scale drug-SE association database that we have built provides an invaluable data resource for computational drug target discovery and drug repositioning.

RX is funded by Case Western Reserve University/Cleveland Clinic CTSA Grant (UL1 RR024989), the Eunice Kennedy Shriver National Institute Of Child Health & Human Development of the National Institutes of Health under Award Number DP2HD084068, the Training grant in Computational Genomic Epidemiology of Cancer (CoGE) (R25 CA094186-06), and Grant #IRG-91-022-18 to the Case Comprehensive Cancer Center from the American Cancer Society. QW is partly funded by ThinTek LLC.