- Open Access
Supervised prediction of drug-induced nephrotoxicity based on interleukin-6 and -8 expression levels
© Su et al.; licensee BioMed Central Ltd. 2014
- Published: 8 December 2014
Drug-induced nephrotoxicity causes acute kidney injury and chronic kidney diseases, and is a major reason for late-stage failures in the clinical trials of new drugs. Therefore, early, pre-clinical prediction of nephrotoxicity could help to prioritize drug candidates for further evaluations, and increase the success rates of clinical trials. Recently, an in vitro model for predicting renal-proximal-tubular-cell (PTC) toxicity based on the expression levels of two inflammatory markers, interleukin (IL)-6 and -8, has been described. However, this and other existing models usually use linear and manually determined thresholds to predict nephrotoxicity. Automated machine learning algorithms may improve these models, and produce more accurate and unbiased predictions.
Here, we report a systematic comparison of the performances of four supervised classifiers, namely random forest, support vector machine, k-nearest-neighbor and naive Bayes classifiers, in predicting PTC toxicity based on IL-6 and -8 expression levels. Using a dataset of human primary PTCs treated with 41 well-characterized compounds that are toxic or not toxic to PTC, we found that random forest classifiers have the highest cross-validated classification performance (mean balanced accuracy = 87.8%, sensitivity = 89.4%, and specificity = 85.9%). Furthermore, we also found that IL-8 is more predictive than IL-6, but a combination of both markers gives higher classification accuracy. Finally, we also show that random forest classifiers trained automatically on the whole dataset have higher mean balanced accuracy than a previous threshold-based classifier constructed for the same dataset (99.3% vs. 80.7%).
Our results suggest that a random forest classifier can be used to automatically predict drug-induced PTC toxicity based on the expression levels of IL-6 and -8.
- supervised classification
- drug toxicity
- random forest
The kidney plays an important role in the maintenance of water and electrolyte balance, and the filtration and elimination of metabolic wastes and drugs from the plasma . Due to drug exposure and active transport and metabolism of drugs, the kidney is susceptible to drug-induced toxicity [2–5]. Nephrotoxic drugs may perturb renal perfusion, induce loss of filtration capacity, and cause damage to the vascular, tubular, glomerular and interstitial cells in the kidney . Drug-induced nephrotoxicity can lead to acute kidney injury, or chronic kidney disease that may process to end-stage kidney disease [6–8]. However, nephrotoxicity of drug candidates is often detected only during the late phases of drug development, and accounts for 19% of drug attrition in phase 3 of clinical trials . Therefore, early, pre-clinical prediction of nephrotoxicity could help to prioritize drug candidates for further evaluations, increase the success rates of clinical trials, and reduce the overall time and cost of drug development.
Renal proximal tubular cells (PTCs) are a major target for drug-induced toxicity because they are involved in the regulation of filtrate concentration and drug transportation and metabolism . Current pre-clinical, in vitro nephrotoxicity predictors are usually based on protein- or gene-expression markers of immortalized renal proximal tubular cell lines [2, 10–13]. Most of these predictors have only been tested in around ten or less compounds . Recently, Li et al. have developed an in vitro predictor based on the expression levels of two inflammatory markers, interleukin (IL)-6 and -8, in human primary renal proximal tubular cells (HPTCs)  and human embryonic stem cell-derived HPTC-like cells . These markers were tested in a larger number of 41 compounds, and gave higher prediction accuracy than many previous predictors . However, most of these existing predictors use simple linear thresholds to distinguish between the effects of nephrotoxic and non-nephrotoxic compounds, even though more than one markers (or "features") are measured from the cells. These manually-determined thresholds may be subject to human biases, and have difficulties in distinguishing features that are non-linearly separable . Therefore, we wonder if non-linear decision boundaries identified automatically using supervised classifiers can further improve the accuracy of nephrotoxicity predictors based on the IL-6 and -8 markers.
Supervised classifier is a computational algorithm that maps, or classifies, input data into different pre-defined categories based on a set of training data whose category membership is known. The support vector machine (SVM) algorithm is one of the most commonly used classifiers. It constructs classification boundaries based on soft margins that allow mis-classified data points, and is especially useful when the data is not linearly separable and/or the number of features is high . The k-nearest-neighbor (k-NN) and naive Bayes classifiers are two other commonly used classifiers. In a k-NN classifier, the category membership of a data point is determined by a majority vote of its neighboring training data points . Naive Bayes classifier is based on the Bayes' theorem and assumes that the measured features are independent . These two classifiers have the advantages of being simple and efficient, especially for low numbers of features [20, 21]. Finally, the random forest algorithm is a relatively new type of classifier based on ensemble learning of a set of decision trees . In certain datasets, random forest may achieve higher classification accuracy than SVM . Despite the popularity of these classifiers, their performances have not been systematically compared and studied under the context of nephrotoxicity prediction.
Here, we report a systematic comparison of random forest, SVM, k-NN, and naive Bayes classifiers in predicting the PTC toxicity of 41 well-characterized compounds that are toxic or not toxic to PTC. The prediction is based on the IL-6 and -8 expression levels measured by Li et al. in HPTCs . We describe how the parameters of all the tested classifiers can be automatically determined without any manual intervention. We also compare the importance of IL-6 and -8 in predicting PTC toxicity. Finally, we also show that supervised prediction based on the best classifier, random forest, achieves higher accuracy than the threshold-based classifier used by Li et al. .
We compared four different supervised classifiers, namely random forest, SVM, k-NN and naive Bayes (Figure 1). The maximum expression levels of IL-8 and -6 induced by the tested compounds were used as inputs to the classifiers. For each classifier, we used a stratified 3-fold cross validation procedure  to estimate its generalized classification performance. This procedure randomly divided the dataset into three roughly equal folds, one of which was used to test a classifier trained on the remaining folds. We repeated the whole cross validation procedure 10 times, each with a different random fold division. The final classification performance was obtained by taking the mean of all the obtained measurements.
where TP is the number of true positives, TN is the number of true negatives, FP is the number of false positives and FN is the number of false negatives. We performed all the analyses using the R statistical environment (v3.0.2) on a personal computer equipped with an Intel Core i7-3770K processor and Windows 7 operating system. The R source code used can be found in Additional File 2.
Random forest is an ensemble learning method that constructs a large number of decision trees (B) during training, and predicts the category label of a new data sample by taking the mode of the labels predicted by these trees . During training, a random subset of m rf features is selected, and the best spit of data points based on these features are used to construct a decision tree. We tested B = 50, 100, 150, 200, 250, 300, 400, 500, 600, 700, 800, 900, 1000, 1200, 1400, 1600, 1800, and 2000. Since our dataset only has two features, we set m rf = 1. The "randomForest" library (v4.6-10) under the R environment was used to perform random forest classification.
Binary support vector machine
The upper bound for the error in the training dataset is provided by , where C is a regularization parameter. This optimization equation allows a trade-off between large margin and small error values.
In our analyses, we tested four SVM kernels, namely linear, polynomial, sigmoid and radial basis function (RBF). We optimized all the parameters, including C, γ, r, and degree of the kernels through an exhaustive grid search . We used the 'e1071' library (v1.6-1) under the R environment to perform SVM classification.
where indicates if belongs to . We tested k = 1, 3, 5, 7 and 9, and used the "class" library (v7.3 - 9) under the R environment to perform k-NN classification.
Naive Bayes classifier
where is the set of feature values in the input data point . We used the 'e1071' library (v1.6-1) under the R environment to perform naive Bayes classification.
Random forest classification
SVM parameter optimization
SVM classification using linear, polynomial, sigmoid and RBF kernels
Classification performance of support vector machines based on different kernels.
Balanced accuracy (%)
Comparison between random forest, SVM, k-NN and naive Bayes classifiers
Classification performance of classifiers trained on both IL-6 and -8 features.
Balanced accuracy (%)
Classification performance of classifiers trained on different combinations of IL-6 and -8 features.
Both IL-6 and -8
Balanced accuracy (%)
Construction of final classifiers using all compounds
Classification performance of final classifiers trained on the whole dataset.
Balanced accuracy (%)
In summary, we have performed a systematic comparison of the performances of four supervised classifiers, namely random forest, SVM, k-NN and naive Bayes classifiers, in predicting nephrotoxicity based on the IL-6 and -8 expression levels. All parameters of the classifiers were determined automatically without any user intervention. We found that random forest classifiers have the highest overall classification performance (mean balanced accuracy = 87.8%, sensitivity = 89.4%, and specificity = 85.9%). Furthermore, we also found that IL-8 is more predictive than IL-6, but a combination of both markers gives higher classification accuracy. Finally, we also show that a final random forest classifier trained automatically on the whole 41-compound dataset has higher classification accuracy than a previous threshold-based classifier  (mean balanced accuracy = 99.3% vs. 80.7%). This better performance is likely due to the non-linear and multivariate decision boundaries generated by the random forest classifier. Our results suggest that a random forest classifier based on these two markers can be used to automatically predict drug-induced nephrotoxicity.
Our methods are general and can be easily applied to test and identify other potential nephrotoxicity markers based on gene expression levels, metabolic profiles, or cellular phenotypes. The classification performance of our classifier may also be further increased by combining markers from these different modalities, and also by increasing the number of training compounds. An important application of our automated classifier is to predict nephrotoxicity of novel chemical compounds identified from large-scale screening of small-molecule or natural product libraries. This will allow early selection and prioritization of compound candidates for further drug development, animal tests or clinical trials, which are costly and time-consuming processes. By focusing on smaller numbers of drug candidates that are less likely to induce nephrotoxicity, the drug or compound discovery process will be more efficient, and the chance of successful clinical trials will also be increased.
The work is supported by a grant from the Joint Council Office (JCO) Development Program, Agency for Science, Technology and Research (A*STAR), Singapore; and the Bioinformatics Institute and the Institute of Bioengineering and Nanotechnology, Biomedical Research Council, A*STAR, Singapore. Publication charges for this work was funded by the Bioinformatics Institute, Biomedical Research Council, A*STAR, Singapore.
This article has been published as part of BMC Bioinformatics Volume 15 Supplement 16, 2014: Thirteenth International Conference on Bioinformatics (InCoB2014): Bioinformatics. The full contents of the supplement are available online at http://www.biomedcentral.com/bmcbioinformatics/supplements/15/S16.
- Lohr JW, Willsky GR, Acara MA: Renal Drug Metabolism. Pharmacol Rev. 1998, 50: 107-142.PubMedGoogle Scholar
- Tiong HY, Huang P, Xiong S, Li Y, Vathsala A, Zink D: Drug-induced nephrotoxicity: clinical impact and pre-clinical in vitro models. Mol Pharm. 2014Google Scholar
- Better Tools for Screening: Early Biomarkers of Kidney Toxicity. [http://www.dddmag.com/articles/2013/12/better-tools-screening-early-biomarkers-kidney-toxicity]
- Bonventre JV, Vaidya VS, Schmouder R, Feig P, Dieterle F: Next-generation biomarkers for detecting kidney toxicity. Nat Biotechnol. 2010, 28: 436-440. 10.1038/nbt0510-436.PubMed CentralView ArticlePubMedGoogle Scholar
- Perazella MA: Renal Vulnerability to Drug Toxicity. Clin J Am Soc Nephrol. 2009, 4: 1275-1283. 10.2215/CJN.02050309.View ArticlePubMedGoogle Scholar
- Choudhury D, Ahmed Z: Drug-associated renal dysfunction and injury. Nat Clin Pract Nephrol. 2006, 2: 80-91.View ArticlePubMedGoogle Scholar
- Levy EM, Viscoli CM, Horwitz RI: The effect of acute renal failure on mortality: A cohort analysis. JAMA. 1996, 275: 1489-1494. 10.1001/jama.1996.03530430033035.View ArticlePubMedGoogle Scholar
- Guo X, Nzerue C: How to prevent, recognize, and treat drug-induced nephrotoxicity. Cleve Clin J Med. 2002, 69: 289-290. 10.3949/ccjm.69.4.289. 293-294, 296-297 passimView ArticlePubMedGoogle Scholar
- Redfern WS, Ewart L, Hammond TG, Bialecki R, Kinter L, Lindgren S, Pollard CE, Roberts R, Rolf MG, Valentin JP: Impact and frequency of different toxicities throughout the pharmaceutical lifecycle. The Toxicologist. 2010, 114: 231-Google Scholar
- Thukral SK, Nordone PJ, Hu R, Sullivan L, Galambos E, Fitzpatrick VD, Healy L, Bass MB, Cosenza ME, Afshari CA: Prediction of Nephrotoxicant Action and Identification of Candidate Toxicity-Related Biomarkers. Toxicol Pathol. 2005, 33: 343-355. 10.1080/01926230590927230.View ArticlePubMedGoogle Scholar
- Waring WS, Moonie A: Earlier recognition of nephrotoxicity using novel biomarkers of acute kidney injury. Clin Toxicol Phila Pa. 2011, 49: 720-728. 10.3109/15563650.2011.615319.View ArticleGoogle Scholar
- Wunnapuk K, Liu X, Peake P, Gobe G, Endre Z, Grice JE, Roberts MS, Buckley NA: Renal biomarkers predict nephrotoxicity after paraquat. Toxicol Lett. 2013, 222: 280-288. 10.1016/j.toxlet.2013.08.003.View ArticlePubMedGoogle Scholar
- Keirstead ND, Wagoner MP, Bentley P, Blais M, Brown C, Cheatham L, Ciaccio P, Dragan Y, Ferguson D, Fikes J, Galvin M, Gupta A, Hale M, Johnson N, Luo W, McGrath F, Pietras M, Price S, Sathe AG, Sasaki JC, Snow D, Walsky RL, Kern G: Early Prediction of Polymyxin-Induced Nephrotoxicity With Next-Generation Urinary Kidney Injury Biomarkers. Toxicol Sci. 2013, kft247-Google Scholar
- Li Y, Oo ZY, Chang SY, Huang P, Eng KG, Zeng JL, Kaestli AJ, Gopalan B, Kandasamy K, Tasnim F, Zink D: An in vitro method for the prediction of renal proximal tubular toxicity in humans. Toxicol Res. 2013, 2: 352-365. 10.1039/c3tx50042j.View ArticleGoogle Scholar
- Li Y, Kandasamy K, Chuah JKC, Lam YN, Toh WS, Oo ZY, Zink D: Identification of Nephrotoxic Compounds with Embryonic Stem-Cell-Derived Human Renal Proximal Tubular-Like Cells. Mol Pharm. 2014Google Scholar
- Burges CJC: A Tutorial on Support Vector Machines for Pattern Recognition. Data Min Knowl Discov. 1998, 2: 121-167. 10.1023/A:1009715923555.View ArticleGoogle Scholar
- Cortes C, Vapnik V: Support-vector networks. Mach Learn. 1995, 20: 273-297.Google Scholar
- Yigit H: A weighting approach for KNN classifier. 2013 Int Conf Electron Comput Comput ICECCO. 2013, 228-231.Google Scholar
- Rish I: An empirical study of the naive Bayes classifier. IJCAI Workshop Empir Methods AI. 2001Google Scholar
- Lepisto L, Kunttu I, Visa A: Rock image classification based on k-nearest neighbour voting. Vis Image Signal Process IEE Proc -. 2006, 153: 475-482. 10.1049/ip-vis:20050315.View ArticleGoogle Scholar
- Kim SB, Han KS, Rim HC, Myaeng SH: Some Effective Techniques for Naive Bayes Text Classification. IEEE Trans Knowl Data Eng. 2006, 18: 1457-1466.View ArticleGoogle Scholar
- Breiman L: Random Forests. Mach Learn. 2001, 45: 5-32. 10.1023/A:1010933404324.View ArticleGoogle Scholar
- Statnikov A, Wang L, Aliferis CF: A comprehensive comparison of random forests and support vector machines for microarray-based cancer classification. BMC Bioinformatics. 2008, 9: 319-10.1186/1471-2105-9-319.PubMed CentralView ArticlePubMedGoogle Scholar
- Araki M, Fahmy N, Zhou L, Kumon H, Krishnamurthi V, Goldfarb D, Modlin C, Flechner S, Novick AC, Fairchild RL: Expression of IL-8 during reperfusion of renal allografts is dependent on ischemic time. Transplantation. 2006, 81: 783-788. 10.1097/01.tp.0000198736.69527.32.View ArticlePubMedGoogle Scholar
- Grigoryev DN, Liu M, Hassoun HT, Cheadle C, Barnes KC, Rabb H: The local and systemic inflammatory transcriptome after acute kidney injury. J Am Soc Nephrol JASN. 2008, 19: 547-558. 10.1681/ASN.2007040469.View ArticlePubMedGoogle Scholar
- Tramma D, Hatzistylianou M, Gerasimou G, Lafazanis V: Interleukin-6 and interleukin-8 levels in the urine of children with renal scarring. Pediatr Nephrol Berl Ger. 2012, 27: 1525-1530. 10.1007/s00467-012-2156-2.View ArticleGoogle Scholar
- Akcay A, Nguyen Q, Edelstein CL: Mediators of inflammation in acute kidney injury. Mediators Inflamm. 2009, 2009: 137072-PubMed CentralView ArticlePubMedGoogle Scholar
- Trevor Hastie, Robert Tibshirani, Jerome Friedman: The Elements of Statistical Learning - Data Mining, Inference, and Prediction. 2009, New York: Springer-Verlag, 2Google Scholar
- Jieping Ye TX: SVM versus Least Squares SVM. J Mach Learn Res - Proc Track. 2007, 2: 644-651.Google Scholar
- Hsu C, Chang C, Lin C: A Practical Guide to Support Vector Classification. 2003Google Scholar
- Müller K-R, Mika S, Rätsch G, Tsuda K, Schölkopf B: An introduction to kernel-based learning algorithms. IEEE Trans NEURAL Netw. 2001, 12: 181-201. 10.1109/72.914517.View ArticlePubMedGoogle Scholar
- Fawcett T: An introduction to ROC analysis. Pattern Recognit Lett. 2006, 27: 861-874. 10.1016/j.patrec.2005.10.010.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.