MRMPlus: an open source quality control and assessment tool for SRM/MRM assay development
© Aiyetan et al. 2015
Received: 1 July 2015
Accepted: 3 December 2015
Published: 12 December 2015
Selected and multiple reaction monitoring involves monitoring a multiplexed assay of proteotypic peptides and associated transitions in mass spectrometry runs. To describe peptide and associated transitions as stable, quantifiable, and reproducible representatives of proteins of interest, experimental and analytical validation is required. However, inadequate and disparate analytical tools and validation methods predispose assay performance measures to errors and inconsistencies.
Implemented as a freely available, open-source tool in the platform independent Java programing language, MRMPlus computes analytical measures as recommended recently by the Clinical Proteomics Tumor Analysis Consortium Assay Development Working Group for “Tier 2” assays – that is, non-clinical assays sufficient enough to measure changes due to both biological and experimental perturbations. Computed measures include; limit of detection, lower limit of quantification, linearity, carry-over, partial validation of specificity, and upper limit of quantification.
MRMPlus streamlines assay development analytical workflow and therefore minimizes error predisposition. MRMPlus may also be used for performance estimation for targeted assays not described by the Assay Development Working Group. MRMPlus’ source codes and compiled binaries can be freely downloaded from https://bitbucket.org/paiyetan/mrmplusgui and https://bitbucket.org/paiyetan/mrmplusgui/downloads respectively.
Selected or Multiple reaction monitoring (SRM and MRM) mass spectrometry is the most widely used MS-based targeted proteomic approach. In contrast to discovery proteomics [1–6], targeted proteomics strategies entail limiting the number of features monitored and optimizes chromatography, instrument tuning, and acquisition methods to achieve high sensitivity . Its advantages over traditional low-throughput quantitative approaches such as ELISA, Western blotting, and immunohistochemistry include multiplexing, a precise relative and absolute quantification of endogenous analytes, no antibody requirement, and an ability to detect unmodified and posttranslationally modified forms of proteins [8–13]. These limitations and opportunities presented by SRM/MRM are well articulated by Shi et al. . As surrogates to protein expression level quantification methods such as mRNA microarray and RNASeq methods fall short of an ability to detect posttranslationally modified proteins, the demand for SRM or MRM-based analytical approaches is anticipated to increase .
Similar to parallel reaction monitoring (PRM), MRM typically involves monitoring a multiplexed assay of proteotypic peptides. However under conditions that permit high resolution and high mass accuracy, only a set of preselected transitions are monitored in MRM as opposed to all transitions in PRM [15, 16]. An MRM assay development workflow may be broadly sub-divided into a pre-mass spectrometry acquisition phase and a post-acquisition phase. The pre-acquisition phase includes, 1) generation of target protein list, 2) selection of proteotypic peptides and 3) an experimental design step . The post-acquisition phase recently described by Colangelo et al  entails four major steps; 1) peak detection, integration and quantification, 2) data quality assessment, 3) data visualization and exploratory analysis, and 4) fold change/statistical significance analysis.
Irrespective of the targeted proteomics approach employed, the complexities of media within which monitored proteotypic peptides that represent proteins of interest reside result in the somewhat unpredictable analytical behavior of peptides and transitions. Thus, it is important that experimental and analytical validation is performed to describe peptides and associated transitions as stable, quantifiable, and reproducible representatives of proteins of interest.
To standardize and validate quantifiable targeted mass spectrometry-based peptide and protein quantifications, the National Cancer Institute (NCI) National Institutes of Health through the Clinical Proteomic Tumor Analysis Consortium (CPTAC) Assay Development Working Group (ADWG) recently described guidelines, experiments and analytical measures for MRM assay characterization . Extending previously published guidelines and studies [19, 20], the described guidelines ensure some levels of confidence for assayed peptides and proteins in quantitative targeted proteomics studies.
Although MRMPlus was conceptualized and implemented to compute the performance of assays developed according to guidelines established by the CPTAC Assay Development Working Group, it may also be used for performance calculations for targeted assays not described by the Working Group.
Inputs to the program include Skyline-derived output files in tab-delimited format (Additional files 1 and 4: Tables S1 and S4), a user defined experiment metadata (Additional file 2: Table S2), and a dilution information data file (Additional file 3: Table S3). Current implementation of MRMPlus computes analytical measures for “Response Curve” and “Mini-Validation of Repeatability” experiments according to the ADWG’s current specifications (Additional files 5 and 6: Tables S5 and S6). The response curve serves to identify the lower limit of quantification (LLOQ), the limit of detection (LOD), and the linear range of a peptide of interest in a multipoint serial dilution of a biological replicate or matrix. The mini-validation of repeatability experiments attempt to replicate peptide measurements within a complex mixture over multiple days. Its derived metrics approximate the variability of the assay in real-life practice.
Default runtime initiated interface to MRMPlus accepts inputs to compute “Response Curve” analytical measures and the ‘More Options’ button instantiates the “Mini-Validation of Repeatability” module (Fig. 2). Users may modify the configuration file (MRMPlus.config) in the application package to set runtime default options.
Setting the ‘Visualize Results’ option on the ‘More Options’ panel to ‘TRUE’ instructs MRMPlus to generate performance associated visualizations (Fig. 2). MRMPlus generates 8 visualization panels to provide a global perspective of assay performance across all assayed peptide transitions. In addition, MRMPlus provides peptide-specific (‘Response Curve’ and ‘Repeatability’) performance visualizations. (Additional files 7: Figure S1 and Additional file 8: Figure S2).
The experimental work that was conducted in this study did not require ethics approval. No patient-derived specimens were used to generate the data for the development of MRMPlus. All of the data were derived from formerly N-linked glycopeptides enriched from commercially-available serum and spiked with crude peptides.
Results and discussions
To demonstrate MRMPlus’ capabilities, we employed it to compute the assay performance metrics on our panel of formerly N-linked glycopeptides. Peptides were selected from in-house discovery phase proteomic experiment analyses of prostate cancer tissues and serum studies. MRM assays to these were developed using commercially available serum and heavy labeled synthetic peptides as spiked-ins. (Please see https://assays.cancer.gov/guidance-document/ for detailed description of experiments). Here in this case study, we subset fifteen of the profiled peptides and their respectively associated transitions for MRMPlus’ demonstration purpose.
MRMPlus computes performance estimates on two levels for each targeted peptide: 1) each associated transition level and 2) the summation observed transition values. Computed results are presented in tables in which the rows are performance measures for the respectively assayed peptide transition, and the columns are the computed measures of analytical performance. (Additional file 5: Table S5, and Additional file 6: Table S6). It should be noted that no absolute values are reported when the peptide transition values are summed with respect to ‘PVSpec.maxDeviation’, ‘PVSpec.meanAtMaxDev’, ‘PVSpec.dayAtMaxDev’, and ‘PVSpec.concLevelAtMaxDev’ performance measures, because these are specific to transitions.
For ‘Response Curve’ performances, limit of detection across peptide transitions span several magnitudes with minimum LOD reported as 0.000524, and maximum reported as 2293.62 (Additional file 5: Table S5, ‘LOD.value’ column). In all cases, LOD was assessed using spiked-in concentrations (Additional file 5: Table S5, ‘LOD.usedSpiked’ column) as pre-curve blank mass spectrometry runs reported no detectable levels. This is possibly due to the mass spectrometer's sensitivity or resolving power. A lower of limit of quantification (Additional file 5: Table S5, ‘LLOQ.CaliPoint’ column) could not be estimated for peptide transitions YLGNATAIFFLPDEGK (2.b9.1), AGPNGTLFVADAYK (2.y10.1), and INNTHALVSLLQNLNK (3.b6.1). We observed an almost absolutely linear response across all transitions (Additional file 5: Table S5, ‘Curve.Rsquared’ and ‘Curve.SlopeStdErr(%)’ columns) with minimum standard error percent of the slope of the response curve reported as 0.0030 % and maximum as 4.3979 %.
For ‘Mini-Validation of Repeatability’, MRMPlus reports the intra-assay coefficient of variation, inter-assay coefficient of variation, and the total coefficient of variation for each peptide transition at respectively specified low, medium, and high concentration levels. The lowest total CV at the validated lower limit of quantification 3.3918 was found associated with peptide transition YLGNATAIFFLPDEGK (2.y8.1) (Additional file 6: Table S6, ‘ValidatedLLOQ.concLevel’ and ‘totalCV.Me’ columns).
The CPTAC ADWG recommends that for a transition to be deemed specific to a peptide, in all samples above the lower limit of quantification, no transition ratio (ratios of peak areas of different transitions of same peptide) in each sample at all concentrations should deviate >30 % from the mean. MRMPlus reports the maximum deviation percentage observed for each transition; such that a deviation above the specified 30 % indicates non-specificity of such transition to the associated peptide.
Thomas et al.  – recently published, provides a complete description of a study of ours demonstrating the utility of MRMPlus.
Comparing MRMPlus to PanoramaWeb
We compared MRMPlus’ results with comparable outputs derived from PanoramaWeb  on our data (Additional file 9: Table S7 and Additional file 10: Table S8). MRMPlus and PanoramaWeb derived exactly the same results for estimation of ‘Repeatability’ performance. With respect to ‘Response Curve’ performance estimation MRMPlus’ and PanoramaWeb results varied. These may particularly be due to their differing methods of estimating linearity. However, Both MRMPlus’ and PanoramaWeb report excellent statistical measure of how close their data are fitted to the regression line – indicated by similar R-squared estimation.
MRMPlus facilitates MRM assay development. By aligning its workflow with Skyline, a previously established tool for building, analyzing, and refining targeted mass spectrometry methods and the resulting data, MRMPlus streamlines and simplifies assay performance estimation. Future developments of MRMPlus are anticipated to be integrated to the Skyline open-source tool for an even more streamlined analytical workflow.
Availability and requirements
Operating system: Platform independent
Programming Language: Java
License: The BSD 3-Clause License
Other requirements: Please see documentation here - https://bitbucket.org/paiyetan/mrmplusgui/wiki/Home.
Any restrictions to use by non-academics: None
The authors acknowledge all the members of the Center for Biomarker Discovery and Translation, Johns Hopkins Medical Institution; and members of the Assay Development Working Group (ADWG) of the National Cancer Institute (NCI) Clinical Proteomics Tumor Analysis Consortium (CPTAC). MRMPlus’ development was supported by the National Institutes of Health, National Cancer Institute, Clinical Proteomic Tumor Analysis Consortium (CPTAC, U24CA160036) ; the Early Detection Research Network (EDRN, U01CA152813); and the National Heart, Lung, and Blood Institute, Programs of Excellence in Glycosciences (PEG, P01HL107153). The authors also acknowledge the help of Jeffrey Whiteaker, Ph.D., of the Fred Hutchinson Cancer Research Center in Seattle, Washington, with deriving comparable performance results on PanoramaWeb.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Washburn MP, Wolters D, Yates JR. Large-scale analysis of the yeast proteome by multidimensional protein identification technology. Nat Biotechnol. 2001;19(3):242.View ArticlePubMedGoogle Scholar
- Ünlü M, Morgan ME, Minden JS. Difference gel electrophoresis. A single gel method for detecting changes in protein extracts. Electrophoresis. 1997;18(11):2071–7.View ArticlePubMedGoogle Scholar
- Gygi SP, Rist B, Gerber SA, Turecek F, Gelb MH, Aebersold R. Quantitative analysis of complex protein mixtures using isotope-coded affinity tags. Nat Biotechnol. 1999;17(10):994.View ArticlePubMedGoogle Scholar
- Ross PL, Huang YN, Marchese JN, Williamson B, Parker K, Hattan S, et al. Multiplexed protein quantitation in saccharomyces cerevisiae using amine-reactive isobaric tagging reagents. Mol Cell Proteomics. 2004;3(12):1154–69.View ArticlePubMedGoogle Scholar
- Ong SE, Blagoev B, Kratchmarova I, Kristensen DB, Steen H, Pandey A, et al. Stable isotope labeling by amino acids in cell culture, SILAC, as a simple and accurate approach to expression proteomics. Mol Cell Proteomics. 2002;1(5):376.View ArticlePubMedGoogle Scholar
- Asara JM, Christofk HR, Freimark LM, Cantley LC. A label-free quantification method by MS/MS TIC compared to SILAC and spectral counting in a proteomics screen. Proteomics. 2008;8(5):994.View ArticlePubMedGoogle Scholar
- Colangelo CM, Chung L, Bruce C, Cheung K. Review of software tools for design and analysis of large scale MRM proteomic datasets. Methods. 2013;61(3):287.View ArticlePubMedPubMed CentralGoogle Scholar
- Liebler DC, Zimmerman LJ. Targeted quantitation of proteins by mass spectrometry. Biochemistry (N Y ). 2013;52(22):3797.View ArticleGoogle Scholar
- Lange V, Malmström JA, Didion J, King NL, Johansson BP, Schäfer J, et al. Targeted quantitative analysis of streptococcus pyogenes virulence factors by multiple reaction monitoring. Mol Cell Proteomics. 2008;7(8):1489.View ArticlePubMedPubMed CentralGoogle Scholar
- Pan S, Aebersold R, Chen R, Rush J, Goodlett DR, McIntosh MW, et al. Mass spectrometry based targeted protein quantification: Methods and applications. J Proteome Res. 2009;8(2):787.View ArticlePubMedPubMed CentralGoogle Scholar
- Picotti P, Aebersold R. Selected reaction monitoring-based proteomics: Workflows, potential, pitfalls and future directions. Nat Methods. 2012;9(6):555.View ArticlePubMedGoogle Scholar
- Kondrat RW, McClusky GA, Cooks RG. Multiple reaction monitoring in mass spectrometry/mass spectrometry for direct analysis of complex mixtures. Anal Chem. 1978;50(14):2017–21.View ArticleGoogle Scholar
- Domon B, Aebersold R. Mass spectrometry and protein analysis. Science (New York, NY). 2006;312(5771):212.View ArticleGoogle Scholar
- Shi T, Su D, Liu T, Tang K, Camp DG, Qian WJ, et al. Advancing the sensitivity of selected reaction monitoring‐based targeted quantitative proteomics. Proteomics. 2012;12(8):1074–92.View ArticlePubMedPubMed CentralGoogle Scholar
- Lange V, Picotti P, Domon B, Aebersold R. Selected reaction monitoring for quantitative proteomics: A tutorial. Mol Syst Biol. 2008;4:222.View ArticlePubMedPubMed CentralGoogle Scholar
- Peterson AC, Russell JD, Bailey DJ, Westphall MS, Coon JJ. Parallel reaction monitoring for high resolution and high mass accuracy quantitative, targeted proteomics. Mol Cell Proteomics. 2012;11(11):1475.View ArticlePubMedPubMed CentralGoogle Scholar
- Brusniak MY, Chu CS, Kusebauch U, Sartain MJ, Watts JD, Moritz RL. An assessment of current bioinformatic solutions for analyzing LC-MS data acquired by selected reaction monitoring technology. Proteomics. 2012;12(8):1176.View ArticlePubMedGoogle Scholar
- Whiteaker JR, Halusa GN, Hoofnagle AN, Sharma V, MacLean B, Yan P, et al. CPTAC assay portal: A repository of targeted proteomic assays. Nat Methods. 2014;11(7):703.View ArticlePubMedPubMed CentralGoogle Scholar
- Mani DR, Abbatiello SE, Carr SA. Statistical characterization of multiple-reaction monitoring mass spectrometry (MRM-MS) assays for quantitative proteomics. BMC Bioinformatics. 2012;13:1.View ArticleGoogle Scholar
- Carr SA, Abbatiello SE, Ackermann BL, Borchers C, Domon B, Deutsch EW, et al. Targeted peptide measurements in biology and medicine: Best practices for mass spectrometry-based assay development using a fit-for-purpose approach. Mol Cell Proteomics. 2014;13(3):907.View ArticlePubMedPubMed CentralGoogle Scholar
- Cham Mead JA, Bianco L, Bessant C. Free computational resources for designing selected reaction monitoring transitions. Proteomics. 2010;10(6):1106.View ArticlePubMedGoogle Scholar
- MacLean B, Tomazela DM, Shulman N, Chambers M, Finney GL, Frewen B, et al. Skyline: An open source document editor for creating and analyzing targeted proteomics experiments. Bioinformatics. 2010;26(7):966.View ArticlePubMedPubMed CentralGoogle Scholar
- Math C. The Apache Commons Mathematics Library. 2014.Google Scholar
- Thomas SN, Harlan R, Chen J, Aiyetan P, Liu Y, Sokoll LJ, et al. Multiplexed Targeted Mass Spectrometry-Based Assays for the Quantification of N-Linked Glycosite-Containing Peptides in Serum. Anal Chem. 2015;87(21):10830–8. doi:10.1021/acs.analchem.5b02063. Epub 2015 Oct 21. PubMed.View ArticlePubMedPubMed CentralGoogle Scholar
- Sharma V, Eckels J, Taylor GK, Shulman NJ, Stergachis AB, Joyner SA, et al. Panorama: a targeted proteomics knowledge base. J Proteome Res. 2014;13(9):4205–10.View ArticlePubMedPubMed CentralGoogle Scholar