- Open Access
MRMPlus: an open source quality control and assessment tool for SRM/MRM assay development
BMC Bioinformatics volume 16, Article number: 411 (2015)
Selected and multiple reaction monitoring involves monitoring a multiplexed assay of proteotypic peptides and associated transitions in mass spectrometry runs. To describe peptide and associated transitions as stable, quantifiable, and reproducible representatives of proteins of interest, experimental and analytical validation is required. However, inadequate and disparate analytical tools and validation methods predispose assay performance measures to errors and inconsistencies.
Implemented as a freely available, open-source tool in the platform independent Java programing language, MRMPlus computes analytical measures as recommended recently by the Clinical Proteomics Tumor Analysis Consortium Assay Development Working Group for “Tier 2” assays – that is, non-clinical assays sufficient enough to measure changes due to both biological and experimental perturbations. Computed measures include; limit of detection, lower limit of quantification, linearity, carry-over, partial validation of specificity, and upper limit of quantification.
MRMPlus streamlines assay development analytical workflow and therefore minimizes error predisposition. MRMPlus may also be used for performance estimation for targeted assays not described by the Assay Development Working Group. MRMPlus’ source codes and compiled binaries can be freely downloaded from https://bitbucket.org/paiyetan/mrmplusgui and https://bitbucket.org/paiyetan/mrmplusgui/downloads respectively.
Selected or Multiple reaction monitoring (SRM and MRM) mass spectrometry is the most widely used MS-based targeted proteomic approach. In contrast to discovery proteomics [1–6], targeted proteomics strategies entail limiting the number of features monitored and optimizes chromatography, instrument tuning, and acquisition methods to achieve high sensitivity . Its advantages over traditional low-throughput quantitative approaches such as ELISA, Western blotting, and immunohistochemistry include multiplexing, a precise relative and absolute quantification of endogenous analytes, no antibody requirement, and an ability to detect unmodified and posttranslationally modified forms of proteins [8–13]. These limitations and opportunities presented by SRM/MRM are well articulated by Shi et al. . As surrogates to protein expression level quantification methods such as mRNA microarray and RNASeq methods fall short of an ability to detect posttranslationally modified proteins, the demand for SRM or MRM-based analytical approaches is anticipated to increase .
Similar to parallel reaction monitoring (PRM), MRM typically involves monitoring a multiplexed assay of proteotypic peptides. However under conditions that permit high resolution and high mass accuracy, only a set of preselected transitions are monitored in MRM as opposed to all transitions in PRM [15, 16]. An MRM assay development workflow may be broadly sub-divided into a pre-mass spectrometry acquisition phase and a post-acquisition phase. The pre-acquisition phase includes, 1) generation of target protein list, 2) selection of proteotypic peptides and 3) an experimental design step . The post-acquisition phase recently described by Colangelo et al  entails four major steps; 1) peak detection, integration and quantification, 2) data quality assessment, 3) data visualization and exploratory analysis, and 4) fold change/statistical significance analysis.
Irrespective of the targeted proteomics approach employed, the complexities of media within which monitored proteotypic peptides that represent proteins of interest reside result in the somewhat unpredictable analytical behavior of peptides and transitions. Thus, it is important that experimental and analytical validation is performed to describe peptides and associated transitions as stable, quantifiable, and reproducible representatives of proteins of interest.
To standardize and validate quantifiable targeted mass spectrometry-based peptide and protein quantifications, the National Cancer Institute (NCI) National Institutes of Health through the Clinical Proteomic Tumor Analysis Consortium (CPTAC) Assay Development Working Group (ADWG) recently described guidelines, experiments and analytical measures for MRM assay characterization . Extending previously published guidelines and studies [19, 20], the described guidelines ensure some levels of confidence for assayed peptides and proteins in quantitative targeted proteomics studies.
Currently available commercial, proprietary tools and open source, vendor-neutral analytical tools for post-data acquisition processing are extensively described in Cham et al , Brusniak et al , Colangelo et al  and Mani et al . Though these tools are highly credible in addressing individual analytical challenges, we found no suitable standalone, platform-independent tool that readily implements the CPTAC ADWG’s recommendation for the evaluation of assay performance. However, Skyline , described as the most complete open source platform addressing a great deal of the analytical requirements of an MRM assay development protocol, provided a sufficient basis for the development of MRMPlus. Streamlining workflow (Fig. 1) and minimizing error predisposition, MRMPlus takes as input, Skyline derived preprocessed files (Additional file 1: Table S1), in addition to user defined metadata files (Additional files 2 and 3: Tables S2 and S3) to compute analytical and validation measures as recommended and described in the CPTAC ADWG published guidelines (https://assays.cancer.gov/guidance-document/).
Although MRMPlus was conceptualized and implemented to compute the performance of assays developed according to guidelines established by the CPTAC Assay Development Working Group, it may also be used for performance calculations for targeted assays not described by the Working Group.
We implemented MRMPlus in the platform-independent Java programing language (Fig. 2). For statistical evaluations, we utilized classes defined in the Apache Commons Mathematics library . For visualizations, we utilized Java classes defined in the JFreeChart library [http://www.jfree.org/jfreechart/].
Inputs to the program include Skyline-derived output files in tab-delimited format (Additional files 1 and 4: Tables S1 and S4), a user defined experiment metadata (Additional file 2: Table S2), and a dilution information data file (Additional file 3: Table S3). Current implementation of MRMPlus computes analytical measures for “Response Curve” and “Mini-Validation of Repeatability” experiments according to the ADWG’s current specifications (Additional files 5 and 6: Tables S5 and S6). The response curve serves to identify the lower limit of quantification (LLOQ), the limit of detection (LOD), and the linear range of a peptide of interest in a multipoint serial dilution of a biological replicate or matrix. The mini-validation of repeatability experiments attempt to replicate peptide measurements within a complex mixture over multiple days. Its derived metrics approximate the variability of the assay in real-life practice.
Default runtime initiated interface to MRMPlus accepts inputs to compute “Response Curve” analytical measures and the ‘More Options’ button instantiates the “Mini-Validation of Repeatability” module (Fig. 2). Users may modify the configuration file (MRMPlus.config) in the application package to set runtime default options.
Setting the ‘Visualize Results’ option on the ‘More Options’ panel to ‘TRUE’ instructs MRMPlus to generate performance associated visualizations (Fig. 2). MRMPlus generates 8 visualization panels to provide a global perspective of assay performance across all assayed peptide transitions. In addition, MRMPlus provides peptide-specific (‘Response Curve’ and ‘Repeatability’) performance visualizations. (Additional files 7: Figure S1 and Additional file 8: Figure S2).
The experimental work that was conducted in this study did not require ethics approval. No patient-derived specimens were used to generate the data for the development of MRMPlus. All of the data were derived from formerly N-linked glycopeptides enriched from commercially-available serum and spiked with crude peptides.
Results and discussions
To demonstrate MRMPlus’ capabilities, we employed it to compute the assay performance metrics on our panel of formerly N-linked glycopeptides. Peptides were selected from in-house discovery phase proteomic experiment analyses of prostate cancer tissues and serum studies. MRM assays to these were developed using commercially available serum and heavy labeled synthetic peptides as spiked-ins. (Please see https://assays.cancer.gov/guidance-document/ for detailed description of experiments). Here in this case study, we subset fifteen of the profiled peptides and their respectively associated transitions for MRMPlus’ demonstration purpose.
The current implementation of MRMPlus outputs results for ADWG-defined experiments’ ‘Response Curve’ and ‘Mini-Validation of Repeatability’. For ‘Response Curve’ estimations, our assay experiments were performed in triplicates over seven dilution ranges (Fig. 3). For ‘Mini-validation of Repeatability’ assay experiments were similarly performed in triplicates, at three dilution (concentration) levels – “High”, “Medium”, and “Low”. These were repeated on five different days (Fig. 4). In both ‘Response Curve’ and ‘Mini-validation of Repeatability’, we assessed three transitions for each peptide.
MRMPlus computes performance estimates on two levels for each targeted peptide: 1) each associated transition level and 2) the summation observed transition values. Computed results are presented in tables in which the rows are performance measures for the respectively assayed peptide transition, and the columns are the computed measures of analytical performance. (Additional file 5: Table S5, and Additional file 6: Table S6). It should be noted that no absolute values are reported when the peptide transition values are summed with respect to ‘PVSpec.maxDeviation’, ‘PVSpec.meanAtMaxDev’, ‘PVSpec.dayAtMaxDev’, and ‘PVSpec.concLevelAtMaxDev’ performance measures, because these are specific to transitions.
For ‘Response Curve’ performances, limit of detection across peptide transitions span several magnitudes with minimum LOD reported as 0.000524, and maximum reported as 2293.62 (Additional file 5: Table S5, ‘LOD.value’ column). In all cases, LOD was assessed using spiked-in concentrations (Additional file 5: Table S5, ‘LOD.usedSpiked’ column) as pre-curve blank mass spectrometry runs reported no detectable levels. This is possibly due to the mass spectrometer's sensitivity or resolving power. A lower of limit of quantification (Additional file 5: Table S5, ‘LLOQ.CaliPoint’ column) could not be estimated for peptide transitions YLGNATAIFFLPDEGK (2.b9.1), AGPNGTLFVADAYK (2.y10.1), and INNTHALVSLLQNLNK (3.b6.1). We observed an almost absolutely linear response across all transitions (Additional file 5: Table S5, ‘Curve.Rsquared’ and ‘Curve.SlopeStdErr(%)’ columns) with minimum standard error percent of the slope of the response curve reported as 0.0030 % and maximum as 4.3979 %.
For ‘Mini-Validation of Repeatability’, MRMPlus reports the intra-assay coefficient of variation, inter-assay coefficient of variation, and the total coefficient of variation for each peptide transition at respectively specified low, medium, and high concentration levels. The lowest total CV at the validated lower limit of quantification 3.3918 was found associated with peptide transition YLGNATAIFFLPDEGK (2.y8.1) (Additional file 6: Table S6, ‘ValidatedLLOQ.concLevel’ and ‘totalCV.Me’ columns).
The CPTAC ADWG recommends that for a transition to be deemed specific to a peptide, in all samples above the lower limit of quantification, no transition ratio (ratios of peak areas of different transitions of same peptide) in each sample at all concentrations should deviate >30 % from the mean. MRMPlus reports the maximum deviation percentage observed for each transition; such that a deviation above the specified 30 % indicates non-specificity of such transition to the associated peptide.
Thomas et al.  – recently published, provides a complete description of a study of ours demonstrating the utility of MRMPlus.
Comparing MRMPlus to PanoramaWeb
We compared MRMPlus’ results with comparable outputs derived from PanoramaWeb  on our data (Additional file 9: Table S7 and Additional file 10: Table S8). MRMPlus and PanoramaWeb derived exactly the same results for estimation of ‘Repeatability’ performance. With respect to ‘Response Curve’ performance estimation MRMPlus’ and PanoramaWeb results varied. These may particularly be due to their differing methods of estimating linearity. However, Both MRMPlus’ and PanoramaWeb report excellent statistical measure of how close their data are fitted to the regression line – indicated by similar R-squared estimation.
MRMPlus facilitates MRM assay development. By aligning its workflow with Skyline, a previously established tool for building, analyzing, and refining targeted mass spectrometry methods and the resulting data, MRMPlus streamlines and simplifies assay performance estimation. Future developments of MRMPlus are anticipated to be integrated to the Skyline open-source tool for an even more streamlined analytical workflow.
Availability and requirements
Operating system: Platform independent
Programming Language: Java
License: The BSD 3-Clause License
Other requirements: Please see documentation here - https://bitbucket.org/paiyetan/mrmplusgui/wiki/Home.
Any restrictions to use by non-academics: None
Washburn MP, Wolters D, Yates JR. Large-scale analysis of the yeast proteome by multidimensional protein identification technology. Nat Biotechnol. 2001;19(3):242.
Ünlü M, Morgan ME, Minden JS. Difference gel electrophoresis. A single gel method for detecting changes in protein extracts. Electrophoresis. 1997;18(11):2071–7.
Gygi SP, Rist B, Gerber SA, Turecek F, Gelb MH, Aebersold R. Quantitative analysis of complex protein mixtures using isotope-coded affinity tags. Nat Biotechnol. 1999;17(10):994.
Ross PL, Huang YN, Marchese JN, Williamson B, Parker K, Hattan S, et al. Multiplexed protein quantitation in saccharomyces cerevisiae using amine-reactive isobaric tagging reagents. Mol Cell Proteomics. 2004;3(12):1154–69.
Ong SE, Blagoev B, Kratchmarova I, Kristensen DB, Steen H, Pandey A, et al. Stable isotope labeling by amino acids in cell culture, SILAC, as a simple and accurate approach to expression proteomics. Mol Cell Proteomics. 2002;1(5):376.
Asara JM, Christofk HR, Freimark LM, Cantley LC. A label-free quantification method by MS/MS TIC compared to SILAC and spectral counting in a proteomics screen. Proteomics. 2008;8(5):994.
Colangelo CM, Chung L, Bruce C, Cheung K. Review of software tools for design and analysis of large scale MRM proteomic datasets. Methods. 2013;61(3):287.
Liebler DC, Zimmerman LJ. Targeted quantitation of proteins by mass spectrometry. Biochemistry (N Y ). 2013;52(22):3797.
Lange V, Malmström JA, Didion J, King NL, Johansson BP, Schäfer J, et al. Targeted quantitative analysis of streptococcus pyogenes virulence factors by multiple reaction monitoring. Mol Cell Proteomics. 2008;7(8):1489.
Pan S, Aebersold R, Chen R, Rush J, Goodlett DR, McIntosh MW, et al. Mass spectrometry based targeted protein quantification: Methods and applications. J Proteome Res. 2009;8(2):787.
Picotti P, Aebersold R. Selected reaction monitoring-based proteomics: Workflows, potential, pitfalls and future directions. Nat Methods. 2012;9(6):555.
Kondrat RW, McClusky GA, Cooks RG. Multiple reaction monitoring in mass spectrometry/mass spectrometry for direct analysis of complex mixtures. Anal Chem. 1978;50(14):2017–21.
Domon B, Aebersold R. Mass spectrometry and protein analysis. Science (New York, NY). 2006;312(5771):212.
Shi T, Su D, Liu T, Tang K, Camp DG, Qian WJ, et al. Advancing the sensitivity of selected reaction monitoring‐based targeted quantitative proteomics. Proteomics. 2012;12(8):1074–92.
Lange V, Picotti P, Domon B, Aebersold R. Selected reaction monitoring for quantitative proteomics: A tutorial. Mol Syst Biol. 2008;4:222.
Peterson AC, Russell JD, Bailey DJ, Westphall MS, Coon JJ. Parallel reaction monitoring for high resolution and high mass accuracy quantitative, targeted proteomics. Mol Cell Proteomics. 2012;11(11):1475.
Brusniak MY, Chu CS, Kusebauch U, Sartain MJ, Watts JD, Moritz RL. An assessment of current bioinformatic solutions for analyzing LC-MS data acquired by selected reaction monitoring technology. Proteomics. 2012;12(8):1176.
Whiteaker JR, Halusa GN, Hoofnagle AN, Sharma V, MacLean B, Yan P, et al. CPTAC assay portal: A repository of targeted proteomic assays. Nat Methods. 2014;11(7):703.
Mani DR, Abbatiello SE, Carr SA. Statistical characterization of multiple-reaction monitoring mass spectrometry (MRM-MS) assays for quantitative proteomics. BMC Bioinformatics. 2012;13:1.
Carr SA, Abbatiello SE, Ackermann BL, Borchers C, Domon B, Deutsch EW, et al. Targeted peptide measurements in biology and medicine: Best practices for mass spectrometry-based assay development using a fit-for-purpose approach. Mol Cell Proteomics. 2014;13(3):907.
Cham Mead JA, Bianco L, Bessant C. Free computational resources for designing selected reaction monitoring transitions. Proteomics. 2010;10(6):1106.
MacLean B, Tomazela DM, Shulman N, Chambers M, Finney GL, Frewen B, et al. Skyline: An open source document editor for creating and analyzing targeted proteomics experiments. Bioinformatics. 2010;26(7):966.
Math C. The Apache Commons Mathematics Library. 2014.
Thomas SN, Harlan R, Chen J, Aiyetan P, Liu Y, Sokoll LJ, et al. Multiplexed Targeted Mass Spectrometry-Based Assays for the Quantification of N-Linked Glycosite-Containing Peptides in Serum. Anal Chem. 2015;87(21):10830–8. doi:10.1021/acs.analchem.5b02063. Epub 2015 Oct 21. PubMed.
Sharma V, Eckels J, Taylor GK, Shulman NJ, Stergachis AB, Joyner SA, et al. Panorama: a targeted proteomics knowledge base. J Proteome Res. 2014;13(9):4205–10.
The authors acknowledge all the members of the Center for Biomarker Discovery and Translation, Johns Hopkins Medical Institution; and members of the Assay Development Working Group (ADWG) of the National Cancer Institute (NCI) Clinical Proteomics Tumor Analysis Consortium (CPTAC). MRMPlus’ development was supported by the National Institutes of Health, National Cancer Institute, Clinical Proteomic Tumor Analysis Consortium (CPTAC, U24CA160036) ; the Early Detection Research Network (EDRN, U01CA152813); and the National Heart, Lung, and Blood Institute, Programs of Excellence in Glycosciences (PEG, P01HL107153). The authors also acknowledge the help of Jeffrey Whiteaker, Ph.D., of the Fred Hutchinson Cancer Research Center in Seattle, Washington, with deriving comparable performance results on PanoramaWeb.
The authors have declared no financial/commercial conflict of interest.
PA and SNT proposed the tool. PA wrote the program codes and performed analysis. SNT performed the experiments. PA and SNT tested the tool. ZZ and HZ supervised. PA drafted the manuscript and all authors reviewed and approved manuscript.
Sample Skyline-derived response curve experiment input to MRMPlus. (TXT 128 kb)
Sample user-defined response experiment metadata file. (TXT 1 kb)
Sample user-defined dilution file. (TXT 130 bytes)
Sample Skyline-derived repeatability experiment input to MRMPlus. (TXT 199 kb)
Sample MRMPlus computed assay performance metrics for the response curve experiments. (TXT 28 kb)
Sample MRMPlus computed assay performance metrics for assay repeatability experiments. (TXT 13 kb)
MRMPlus visualizations. (PNG 176 kb)
MRMPlus visualization showing a sample global performance view across all assayed peptide transitions. The x-axis represents individual peptide transitions (including summed transitions). The y-axis represent the computed coefficient of variations (inter-assay, intra-assay, and total) at a median level concentration. (PNG 67 kb)
Comparing and contrasting ‘Repeatability’ measures between MRMPlus and PanoramaWeb. (XLSX 9 kb)
Comparing and contrasting ‘Response’ measures between MRMPlus and PanoramaWeb. (XLSX 9 kb)
About this article
Cite this article
Aiyetan, P., Thomas, S.N., Zhang, Z. et al. MRMPlus: an open source quality control and assessment tool for SRM/MRM assay development. BMC Bioinformatics 16, 411 (2015). https://doi.org/10.1186/s12859-015-0838-z
- Assay Performance
- Assay Development
- Peptide Transition
- Proteotypic Peptide
- Minimum Standard Error