SAMPLEX: Automatic mapping of perturbed and unperturbed regions of proteins and complexes
© Krzeminski et al; licensee BioMed Central Ltd. 2010
Received: 19 August 2009
Accepted: 26 January 2010
Published: 26 January 2010
The activity of proteins within the cell is characterized by their motions, flexibility, interactions or even the particularly intriguing case of partially unfolded states. In the last two cases, a part of the protein is affected either by binding or unfolding and the detection of the respective perturbed and unperturbed region(s) is a fundamental part of the structural characterization of these states. This can be achieved by comparing experimental data of the same protein in two different states (bound/unbound, folded/unfolded). For instance, measurements of chemical shift perturbations (CSPs) from NMR 1H-15N HSQC experiments gives an excellent opportunity to discriminate both moieties.
We describe an innovative, automatic and unbiased method to distinguish perturbed and unperturbed regions in a protein existing in two distinct states (folded/partially unfolded, bound/unbound). The SAMPLEX program takes as input a set of data and the corresponding three-dimensional structure and returns the confidence for each residue to be in a perturbed or unperturbed state. Its performance is demonstrated for different applications including the prediction of disordered regions in partially unfolded proteins and of interacting regions in protein complexes.
The proposed approach is suitable for partially unfolded states of proteins, local perturbations due to small ligands and protein-protein interfaces. The method is not restricted to NMR data, but is generic and can be applied to a wide variety of information.
During the last decennia, there has been a growing interest in biological system dynamics, which consist for instance of interactions between proteins and folding pathways. This also includes proteins that partially unfold under specific environmental stimuli (taxis), and act as intermediates in a cascade of events . In all cases, only a part of the protein is involved in the biological process while the remaining part stays dormant. Indeed, in a protein-protein complex, only the interaction surface is often affected, even though, sometimes, a larger part of the protein is modified, like in the case of allosteric interactions or conformational changes upon binding. As for partially unfolded proteins, which are typically composed of mobile regions linked to a folded core, they show some flexibility due to the internal motions of the protein and the reorganization of the unfolded part upon partial unfolding. In order to structurally characterize such systems, it is crucial to distinguish the unperturbed regions from the perturbed ones.
Nuclear Magnetic Resonance (NMR) is particularly convenient for monitoring, at atomic level, structural and/or environmental changes, which occur upon binding to a partner molecule or at (partial) unfolding. In particular, heteronuclear shift correlation experiments, such as 1H-15N HSQC, are very useful to detect alterations in the electronic environment of atoms which affect their chemical shifts. Hence, comparison of HSQC spectra of a molecule in two different states (bound/unbound, folded/unfolded) allows the detection of affected residues by analyzing the chemical shift perturbations (CSPs) [2–4].
A common method to distinguish unperturbed residues from others is to reject the ones for which the CSP is higher than a pre-defined threshold that usually corresponds to the average of all CSPs plus one or two times their standard deviation. This procedure is repeated with the remaining residues, until no rejection occurs anymore. The choice of a cut-off can be biased by subjectivity in the selection process. In addition, there is no simple relationship between the amount of chemical shift perturbations and the magnitude of the perturbation; this sometimes results in biased data (like very low CSP for a residue that is flanked by two others with very high CSPs), leading to misinterpretations. With that respect, it might be more relevant to consider not only the CSP of the concerned residue or its sequential neighbors, but also the CSPs of the residues close to it in the 3D structure.
This problem has previously been addressed by Kalbitzer et al.  for the case of protein-protein interactions. Their method yields the probability for a given residue to be involved in the interaction by comparing its CSP and the average CSP of the same residue found in all complexes of the BMRB database . This approach however depends on the way the CSP is calculated  and an user-defined cut-off. In the present study we present an automated and less biased procedure to discriminate perturbed regions from unperturbed ones in a protein using its three dimensional structure and a set of experimental data. We will use in this work CSP data, but the method is generic and can be used for other types of data as well. The program we developed for this purpose, SAMPLEX (Smoothed Automatic Mapping of Protein from Listed Extremes), is based on a topologic approach and can overcome problems due to peak overlap or to sparse data. We tested our method on several systems, including protein complexes and partially unfolded proteins, for which NMR data were available.
where ΔH N , ΔN, ΔC α and ΔC O are the chemical shift differences of the hydrogen (HN), the amide nitrogen (N), the alpha carbon (Cα) and the carbon of the carboxyl group (CO) of the backbone, respectively, and SW N , SW Cα and SW CO are the spectral width ratio between HN and N, HN and Cα and HN and CO, respectively .
From these CSPs determined for each residue (ideally) and the known native structure of the protein of interest, SAMPLEX automatically delimits the perturbed regions of the protein. The method has been divided into four consecutive steps that aim at defining groups of residues that belong to a similar state (perturbed/unperturbed) within the protein.
Step 1. Attribution of confidences
where CSP i is the CSP of the residue i, μ the average of all CSPs and σ the standard deviation. The factor k can be positive or negative. We subsequently transform these factors into values ρ i such that the highest CSP value(s) becomes 1, the lowest -1 and all others have intermediate values. From now on, we will call ρ i the confidence of the residue i. However, to avoid that the highest and lowest CSP values, which can be extreme, dominate this transformation, we first determine minimum factors k high and k low . For this, we run several times (roughly 1000 times the number of available data) the following process:
Selection of a sub-ensemble made of the highest, the lowest and randomly selected CSP values such that the size of this sub-ensemble corresponds to 15% of the number of available data.
Calculation of the factor k for this sub-ensemble using the equation 1.
We then define k high and k low as the lowest and highest k values found among all runs performed.
Initialization of the variable α ι = 0.
Selection of a sub-ensemble made of the CSP value of residue i and randomly selected CSP values such that the size of this sub-ensemble corresponds to 15% of the number of available data.
Calculation of the factor k i for this given ensemble.
Steps 2 to 4 are repeated n_trial = 100 times the number of available data.
The confidence ρ i of residue i is finally given by the quotient of α i over n_trial.
Step 2. Inference of missing residues
At least two neighboring residues with available experimental data should be detected otherwise the residue is excluded from further calculations and no decision about its state is made in the end. When an ensemble of structures is provided, neighbors are defined by considering the average distance over all models. This allows taking into account possible conformational heterogeneity into the procedure.
Step 3. Homogenization of confidences
In this formula, is the newly attributed confidence of residue i, the starting confidence, n the number of neighbors of residue i, and φ ij the sigmoidal function defined in Eq. 5.
Step 4. Final decision
From the homogenized confidences, we define all residues with a confidence higher than 0.05 as being in a perturbed state and all residues with a confidence lower than -0.05 as remaining in their unperturbed state. The residues in between those two values are considered to be in an ambiguous state, which can mean that they might be close to a perturbed region or indirectly affected by a perturbed region.
Results and Discussion
The selection approach described above was tested on five distinct systems with different characteristics. The various test cases are representative of both binding - and partial unfolding - induced CSP data. In the case of protein-protein complexes, the situation is delicate in the sense that the measure of the accuracy of the selection depends on the way we define the interface and on the quality of the structures. In this work, we defined the residues at the interface as the ones having at least one atom of the backbone within 5 Å of any atom of the other subunit, excluding hydrogen atoms. In the case of an ensemble of conformations, we consider that a contact is effective if it is found in at least one model. Graphical results are given for each test case [Additional file 1].
Performance of diverse test cases
Chymotrypsin inhibitor 2 (CI2) subunit in complex with the Subtilisin BPN'
For BPN', the situation is more complicated because of the size of the protein (281 residues) and the lack of data (16% of confidences had to be inferred). Residues 33, 97, 99-109, 126-128, 141, 154-156, 167-171 and 218-219 are predicted by SAMPLEX as being in a non-native-like environment, and residues 65, 98 and 220 as being in an ambiguous state. The interface measured from the X-ray complex includes residues 99-104, 125-128, 154-157, 167, 218-221. Besides residue 141, all residues selected by SAMPLEX are either in or close to the interface (see figure 2) and consequently could be perturbed due to the interaction with CI2. Residue 141, which is not close to the interface, shows the second highest experimental CSP; this could be due to an indirect perturbation (e.g. conformational change).
The Ubiquitin-conjugating enzyme E2-7 (UBCH5) in complex with CCR4-NOT (CNOT4)
UBCH5 is a human protein involved in the ubiquitination of proteins . It interacts with CNOT4, a transcription factor . From the unbound form of the protein (PDB entry: 2ESO) , our algorithm estimates that residues 1-12, 16-17, 60-64, 88 and 94-101 of UBCH5 are perturbed. No Residue is classified in an ambiguous state. The contacts defined from the structural model obtained with HADDOCK [12, 13] (PDB entry: 1UR6) 8-59, 61-63, 88 and 91 to 96 are involved in the interaction. Although this structural model was obtained based on chemical shift information, we are confident that it presents an accurate picture of the complex due to its high homology to the UbcH7-c-Cbl complex solved by X-ray crystallography (PDB entry 1FBV) .
On the CNOT4 side, the predictions turn out to be rather accurate. The algorithm predicts residues 15 to 19, 41, 45, 47 to 50 and 53 to 57 as perturbed and residues 14, 40, 46 and 52 in an intermediate state. The interface from the structural model comprises residues 15 to 20, 40 to 41, 44 to 45, 48 to 50, 52 to 55 and 57.
Colicin E9 domain (CE9) subunit in complex with its cognate immunity protein (IM9)
CE9 is a small bacterial protein with cytotoxic endonuclease activity. The structures of its unbound and bound forms have been solved by X-ray at 1.80 Å and 1.70 Å, respectively (PDB entries 1FSJ and 1EMW, respectively) . In addition, NMR chemical shifts have been measured for both the unbound and bound forms (BMRB accession numbers 4352 and 4293, respectively) [16, 17]. As for its partner, Im9, both bound (1EMW) and unbound forms are also available. The latter has been solved by NMR (PDB code 1IMP) . Chemical shifts are available for both forms (BMRB accession numbers 4115 and 4116) .
For E9, the perturbed region estimated by SAMPLEX encloses residues 23, 72-79, 86-87 and 94-99. Residues 70, 80, 85 and 93 are in an ambiguous state. From the crystal structure, residues 72-75, 77-, 81, 83, 86, 89 and 97 are at the interface. In the case of Im9, SAMPLEX defines residues 24, 30-37, 47-57 and 82 as perturbed and residue 38 as ambiguous. The crystal structure indicates that residues 23-25, 30, 33-34, 50-51, 54-55 and 62 are involved in the interaction. SAMPLEX predicted a somewhat larger region in both proteins that correctly encompass the true interface. The additional predictions are mainly located in loops that can undergo conformational changes upon binding. Indeed, chemical shift perturbations alone cannot distinguish between direct interactions and induced conformational changes.
The lactose operon repressor and its inducer
The Lac repressor is a tetrameric protein that binds a specific operator on DNA and therefore inactivates the transcription of the enzymes involved in the metabolism of the lactose [20, 21]. When a molecule of isopropyl-β-D-thiogalactoside (IPTG) is embedded in the repressor, this later is inhibited due to a structural rearrangement.
In this case, SAMPLEX predicts residues 66-82, 97, 124, 126-128, 133, 160-163, 192-197, 222, 276-280, 292-293, 319-320 and 322 as perturbed. In addition, residues 83, 95-96, 98-99, 125, 132, 221, 294 and 318 are classified as ambiguous state. The prediction corresponds to the binding pocket of IPTG and the beta sheet involved in the interaction between two monomers, which has previously been demonstrated as being perturbed .
The Photoactive Yellow Protein (PYP)
PYP is a 14 kDa protein found in the bacterium Halorhodospira halophila. Upon illumination at 446 nm, this protein partially unfolds and triggers the signal, which results in the movement by the bacterium away from the light [23–25]. The protein contains a PAS domain as found in many signaling proteins . The chromophore is a p-coumaric acid, which undergoes a trans-to-cis isomerization after activation of the ground state. This isomerization results in the partial unfolding of the protein. A truncated form of PYP, Δ25-PYP, with the 25 first N-terminal amino acids deleted, shows a longer lifetime in the excited state (about 10 minutes), which allowed to record 1H-15N HSQC spectra and compare the ground and excited states .
Using as input the NMR structure of Δ25-PYP (PDB entry 1XFN)  the program defines the regions that comprise residues 29, 42-52, 54-58, 63-65, 67-74, 76-77 and 97 to 102 as perturbed, i.e., in this case, as partially unfolded. Residues 53, 66, 78-79 and 103 are in an ambiguous state. These regions contain the loop, which bears the chromophore (residue 69), residue E46, which is part of helix 3 and yields a proton to the chromophore before unfolding of the protein, and the loop flanked by beta sheets 4 and 5. This loop can be seen as the lead that covers the chromophore. Finally, a part of the helix 5 is perturbed, while all beta sheets stay in their native like environment. In conclusion, only the part surrounding the chromophore is predicted as affected.
Since the structure of the partially unfolded state of Δ25-PYP has been solved by NMR, we investigated the dependence of the predictions on the structure used: when running SAMPLEX from the unfolded structure (PDB code 1XFQ)  residues 28-29, 43-58, 63-65, 67-68, 70-74, 77-78, 97-102 are predicted as perturbed and residues 42, 66, 69, 76, 79, 103 as ambiguous. These results are quite similar (97% overlap considering perturbed and ambiguous regions together) to the ones obtained using the folded protein indicating that our method is robust with respect to the structure used.
An unperturbed protein
SAMPLEX can also assess whether data are indicative of a perturbation or not, by making use of the relative standard deviation σ R defined as σ R = σ/μ, where μ and σ are the mean and the standard deviation of all available data, respectively. Below a value of 25%, the program will consider that no part of the protein has been affected and will ask the user whether he wishes to continue. We tested this by analyzing HSQC spectra of the Lac Headpiece repressor in complex with DNA at two different frequencies (500 MHz and 900 MHz) and calculating CSP values from their difference. The resulting σ R was 5.9% and consequently SAMPLEX did not make any prediction. In the case of the other examples discussed in this paper, σ R increased to 200.7%, 286.4%, 141.6%, 115.5%, 153.0%, 200.8%, 117.6% and 88.9% for CI2, BPN', UBCH5, CNOT4, CE9, Im9, the lactose operon repressor and PYP, respectively.
Comparison with general methods
to consider as perturbed all residues with a CSP value higher than the average of all available data plus one or two times their standard deviation
to iterate a process in which residues with a CSP value higher than the average plus one or two times their standard deviation are rejected and start again with the remaining residues until no rejection occurs; all rejected are finally classified as perturbed.
where TP, FP, TN and FN are the true positive (correctly predicted), false positive (wrongly predicted), true negative (correctly not predicted) and false negative (wrongly not predicted) residues, respectively. This coefficient reaches 1 when the prediction is perfect.
Performance of SAMPLEX
Method A 1
Method B 2
In this paper we have described a new un-biased strategy to distinguish perturbed from unperturbed regions in NMR spectra that define two different states of a protein. The program developed for this purpose, SAMPLEX, requires on the one hand the chemical shifts (or some other kind of data distinguishing the various states) of the protein in both the ground/free and the excited/bound state and on the other hand the structure (or an ensemble of structures) of the ground/free form as input. The method can be used to find the partially unfolded moiety of a protein, or in the case of proteins complexes, to define the interaction surface.
In this work, we only used CSP data from 1H-15N HSQC spectra and showed that they are already sufficient for a successful selection. However, since the result will depend on the quality and the amount of data, SAMPLEX would yield more accurate solution by including additional chemical shifts (13Cα, 13CO...), in particular in the case of large proteins. It is worth noting again that, in the case of complexes, chemical shifts can report on both direct interaction and indirect effects such as remote conformational changes; results should thus always be carefully analyzed.
It should be noted that our selection method is generic and therefore not restricted to NMR CSP data. It can also be applied to e.g. order parameters from NMR relaxation data, protection factors calculated from H/D exchange experiments or any other experimental data as long as it provides information on a molecule at an atomic level. This makes it applicable to a large variety of problems in which some selection/classification needs to be performed.
SAMPLEX is written in python and is available free of charge. It can be downloaded from: http://www.nmr.chem.uu.nl/Software/samplex.
We acknowledge financial support from the Netherlands Organization for Scientific Research (NWO) ("van Molecuul tot Cell" grant n° 805.47.121 and VICI grant n° 700.56.442 to A.M.J.J.B.) and from the European Community (FP6 STREP "UPMAN", contract n° LSHG-CT-2005-512052).
- Horst MA, Hellingwerf KJ: Photoreceptor proteins, "star actors of modern times": a review of the functional dynamics in the structure of representative members of six different photoreceptor families. Acc Chem Res 2004, 37(1):13–20. 10.1021/ar020219dView ArticlePubMedGoogle Scholar
- Farmer BT, Constantine KL, Goldfarb V, Friedrichs MS, Wittekind M, Yanchunas J Jr, Robertson JG, Mueller L: Localizing the NADP+ binding site on the MurB enzyme by NMR. Nat Struct Biol 1996, 3(12):995–997. 10.1038/nsb1296-995View ArticlePubMedGoogle Scholar
- Mulder FA, Schipper D, Bott R, Boelens R: Altered flexibility in the substrate-binding site of related native and engineered high-alkaline Bacillus subtilisins. J Mol Biol 1999, 292(1):111–123. 10.1006/jmbi.1999.3034View ArticlePubMedGoogle Scholar
- Zuiderweg ER: Mapping protein-protein interactions in solution by NMR spectroscopy. Biochemistry 2002, 41(1):1–7. 10.1021/bi011870bView ArticlePubMedGoogle Scholar
- Schumann FH, Riepl H, Maurer T, Gronwald W, Neidig KP, Kalbitzer HR: Combined chemical shift changes and amino acid specific chemical shift mapping of protein-protein interactions. J Biomol NMR 2007, 39(4):275–289. 10.1007/s10858-007-9197-zView ArticlePubMedGoogle Scholar
- Ulrich EL, Akutsu H, Doreleijers JF, Harano Y, Ioannidis YE, Lin J, Livny M, Mading S, Maziuk D, Miller Z, et al.: BioMagResBank. Nucleic Acids Res 2008, (36 Database):D402–408.Google Scholar
- Bott R, Ultsch M, Kossiakoff A, Graycar T, Katz B, Power S: The three-dimensional structure of Bacillus amyloliquefaciens subtilisin at 1.8 A and an analysis of the structural consequences of peroxide inactivation. J Biol Chem 1988, 263(16):7895–7906.PubMedGoogle Scholar
- Radisky ES, Koshland DE Jr: A clogged gutter mechanism for protease inhibitors. Proc Natl Acad Sci USA 2002, 99(16):10316–10321. 10.1073/pnas.112332899View ArticlePubMedPubMed CentralGoogle Scholar
- Scheffner M, Huibregtse JM, Howley PM: Identification of a human ubiquitin-conjugating enzyme that mediates the E6-AP-dependent ubiquitination of p53. Proc Natl Acad Sci USA 1994, 91(19):8797–8801. 10.1073/pnas.91.19.8797View ArticlePubMedPubMed CentralGoogle Scholar
- Albert TK, Lemaire M, van Berkum NL, Gentz R, Collart MA, Timmers HT: Isolation and characterization of human orthologs of yeast CCR4-NOT complex subunits. Nucleic Acids Res 2000, 28(3):809–817. 10.1093/nar/28.3.809View ArticlePubMedPubMed CentralGoogle Scholar
- Ozkan E, Yu H, Deisenhofer J: Mechanistic insight into the allosteric activation of a ubiquitin-conjugating enzyme by RING-type ubiquitin ligases. Proc Natl Acad Sci USA 2005, 102(52):18890–18895. 10.1073/pnas.0509418102View ArticlePubMedPubMed CentralGoogle Scholar
- de Vries SJ, van Dijk AD, Krzeminski M, van Dijk M, Thureau A, Hsu V, Wassenaar T, Bonvin AM: HADDOCK versus HADDOCK: new features and performance of HADDOCK2.0 on the CAPRI targets. Proteins 2007, 69(4):726–733. 10.1002/prot.21723View ArticlePubMedGoogle Scholar
- Dominguez C, Boelens R, Bonvin AM: HADDOCK: a protein-protein docking approach based on biochemical or biophysical information. J Am Chem Soc 2003, 125(7):1731–1737. 10.1021/ja026939xView ArticlePubMedGoogle Scholar
- Zheng N, Wang P, Jeffrey PD, Pavletich NP: Structure of a c-Cbl-UbcH7 complex: RING domain function in ubiquitin-protein ligases. Cell 2000, 102(4):533–539. 10.1016/S0092-8674(00)00057-XView ArticlePubMedGoogle Scholar
- Kuhlmann UC, Pommer AJ, Moore GR, James R, Kleanthous C: Specificity in protein-protein interactions: the structural basis for dual recognition in endonuclease colicin-immunity protein complexes. J Mol Biol 2000, 301(5):1163–1178. 10.1006/jmbi.2000.3945View ArticlePubMedGoogle Scholar
- Whittaker SB, Boetzel R, MacDonald C, Lian LY, James R, Kleanthous C, Moore GR: Assignment of 1H, 13C and 15N signals of the DNase domain of colicin E9. J Biomol NMR 1999, 14(2):201–202. 10.1023/A:1008394407597View ArticlePubMedGoogle Scholar
- Whittaker SB, Czisch M, Wechselberger R, Kaptein R, Hemmings AM, James R, Kleanthous C, Moore GR: Slow conformational dynamics of an endonuclease persist in its complex with its natural protein inhibitor. Protein Sci 2000, 9(4):713–720. 10.1110/ps.9.4.713View ArticlePubMedPubMed CentralGoogle Scholar
- Osborne MJ, Breeze AL, Lian LY, Reilly A, James R, Kleanthous C, Moore GR: Three-dimensional solution structure and 13C nuclear magnetic resonance assignments of the colicin E9 immunity protein Im9. Biochemistry 1996, 35(29):9505–9512. 10.1021/bi960401kView ArticlePubMedGoogle Scholar
- Boetzel R, Czisch M, MacDonald CJ, Kaptein R, Hemmings AM, James R, Kleanthous C, Moore GR: Assignment of 1H, 13C and 15N signals of the inhibitor protein Im9 bound to the DNase domain of colicin E9. J Biomol NMR 1998, 12(4):567–568. 10.1023/A:1008397523780View ArticlePubMedGoogle Scholar
- Daber R, Stayrook S, Rosenberg A, Lewis M: Structural analysis of lac repressor bound to allosteric effectors. J Mol Biol 2007, 370(4):609–619. 10.1016/j.jmb.2007.04.028View ArticlePubMedPubMed CentralGoogle Scholar
- Lewis M, Chang G, Horton NC, Kercher MA, Pace HC, Schumacher MA, Brennan RG, Lu P: Crystal structure of the lactose operon repressor and its complexes with DNA and inducer. Science 1996, 271(5253):1247–1254. 10.1126/science.271.5253.1247View ArticlePubMedGoogle Scholar
- Bell CE, Lewis M: The Lac repressor: a second generation of structural and functional studies. Curr Opin Struct Biol 2001, 11(1):19–25. 10.1016/S0959-440X(00)00180-9View ArticlePubMedGoogle Scholar
- Hellingwerf KJ, Crielaard WC, Joost Teixeira de Mattos M, Hoff WD, Kort R, Verhamme DT, Avignone-Rossa C: Current topics in signal transduction in bacteria. Antonie Van Leeuwenhoek 1998, 74(4):211–227. 10.1023/A:1001738419877View ArticlePubMedGoogle Scholar
- Dux P, Rubinstenn G, Vuister GW, Boelens R, Mulder FA, Hard K, Hoff WD, Kroon AR, Crielaard W, Hellingwerf KJ, et al.: Solution structure and backbone dynamics of the photoactive yellow protein. Biochemistry 1998, 37(37):12689–12699. 10.1021/bi9806652View ArticlePubMedGoogle Scholar
- Rubinstenn G, Vuister GW, Mulder FA, Dux PE, Boelens R, Hellingwerf KJ, Kaptein R: Structural and dynamic changes of photoactive yellow protein during its photocycle in solution. Nat Struct Biol 1998, 5(7):568–570. 10.1038/823View ArticlePubMedGoogle Scholar
- Pellequer JL, Wager-Smith KA, Kay SA, Getzoff ED: Photoactive yellow protein: a structural prototype for the three-dimensional fold of the PAS domain superfamily. Proc Natl Acad Sci USA 1998, 95(11):5884–5890. 10.1073/pnas.95.11.5884View ArticlePubMedPubMed CentralGoogle Scholar
- Bernard C, Houben K, Derix NM, Marks D, Horst MA, Hellingwerf KJ, Boelens R, Kaptein R, van Nuland NA: The solution structure of a transient photoreceptor intermediate: Delta25 photoactive yellow protein. Structure 2005, 13(7):953–962. 10.1016/j.str.2005.04.017View ArticlePubMedGoogle Scholar
- Baldi P, Brunak S, Chauvin Y, Andersen CA, Nielsen H: Assessing the accuracy of prediction algorithms for classification: an overview. Bioinformatics 2000, 16(5):412–424. 10.1093/bioinformatics/16.5.412View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.