Skip to content

Advertisement

You're viewing the new version of our site. Please leave us feedback.

Learn more

BMC Bioinformatics

Open Access

MRMPlus: an open source quality control and assessment tool for SRM/MRM assay development

BMC Bioinformatics201516:411

https://doi.org/10.1186/s12859-015-0838-z

Received: 1 July 2015

Accepted: 3 December 2015

Published: 12 December 2015

Abstract

Background

Selected and multiple reaction monitoring involves monitoring a multiplexed assay of proteotypic peptides and associated transitions in mass spectrometry runs. To describe peptide and associated transitions as stable, quantifiable, and reproducible representatives of proteins of interest, experimental and analytical validation is required. However, inadequate and disparate analytical tools and validation methods predispose assay performance measures to errors and inconsistencies.

Results

Implemented as a freely available, open-source tool in the platform independent Java programing language, MRMPlus computes analytical measures as recommended recently by the Clinical Proteomics Tumor Analysis Consortium Assay Development Working Group for “Tier 2” assays – that is, non-clinical assays sufficient enough to measure changes due to both biological and experimental perturbations. Computed measures include; limit of detection, lower limit of quantification, linearity, carry-over, partial validation of specificity, and upper limit of quantification.

Conclusions

MRMPlus streamlines assay development analytical workflow and therefore minimizes error predisposition. MRMPlus may also be used for performance estimation for targeted assays not described by the Assay Development Working Group. MRMPlus’ source codes and compiled binaries can be freely downloaded from https://bitbucket.org/paiyetan/mrmplusgui and https://bitbucket.org/paiyetan/mrmplusgui/downloads respectively.

Background

Selected or Multiple reaction monitoring (SRM and MRM) mass spectrometry is the most widely used MS-based targeted proteomic approach. In contrast to discovery proteomics [16], targeted proteomics strategies entail limiting the number of features monitored and optimizes chromatography, instrument tuning, and acquisition methods to achieve high sensitivity [7]. Its advantages over traditional low-throughput quantitative approaches such as ELISA, Western blotting, and immunohistochemistry include multiplexing, a precise relative and absolute quantification of endogenous analytes, no antibody requirement, and an ability to detect unmodified and posttranslationally modified forms of proteins [813]. These limitations and opportunities presented by SRM/MRM are well articulated by Shi et al. [14]. As surrogates to protein expression level quantification methods such as mRNA microarray and RNASeq methods fall short of an ability to detect posttranslationally modified proteins, the demand for SRM or MRM-based analytical approaches is anticipated to increase [7].

Similar to parallel reaction monitoring (PRM), MRM typically involves monitoring a multiplexed assay of proteotypic peptides. However under conditions that permit high resolution and high mass accuracy, only a set of preselected transitions are monitored in MRM as opposed to all transitions in PRM [15, 16]. An MRM assay development workflow may be broadly sub-divided into a pre-mass spectrometry acquisition phase and a post-acquisition phase. The pre-acquisition phase includes, 1) generation of target protein list, 2) selection of proteotypic peptides and 3) an experimental design step [17]. The post-acquisition phase recently described by Colangelo et al [7] entails four major steps; 1) peak detection, integration and quantification, 2) data quality assessment, 3) data visualization and exploratory analysis, and 4) fold change/statistical significance analysis.

Irrespective of the targeted proteomics approach employed, the complexities of media within which monitored proteotypic peptides that represent proteins of interest reside result in the somewhat unpredictable analytical behavior of peptides and transitions. Thus, it is important that experimental and analytical validation is performed to describe peptides and associated transitions as stable, quantifiable, and reproducible representatives of proteins of interest.

To standardize and validate quantifiable targeted mass spectrometry-based peptide and protein quantifications, the National Cancer Institute (NCI) National Institutes of Health through the Clinical Proteomic Tumor Analysis Consortium (CPTAC) Assay Development Working Group (ADWG) recently described guidelines, experiments and analytical measures for MRM assay characterization [18]. Extending previously published guidelines and studies [19, 20], the described guidelines ensure some levels of confidence for assayed peptides and proteins in quantitative targeted proteomics studies.

Currently available commercial, proprietary tools and open source, vendor-neutral analytical tools for post-data acquisition processing are extensively described in Cham et al [21], Brusniak et al [17], Colangelo et al [7] and Mani et al [19]. Though these tools are highly credible in addressing individual analytical challenges, we found no suitable standalone, platform-independent tool that readily implements the CPTAC ADWG’s recommendation for the evaluation of assay performance. However, Skyline [22], described as the most complete open source platform addressing a great deal of the analytical requirements of an MRM assay development protocol, provided a sufficient basis for the development of MRMPlus. Streamlining workflow (Fig. 1) and minimizing error predisposition, MRMPlus takes as input, Skyline derived preprocessed files (Additional file 1: Table S1), in addition to user defined metadata files (Additional files 2 and 3: Tables S2 and S3) to compute analytical and validation measures as recommended and described in the CPTAC ADWG published guidelines (https://assays.cancer.gov/guidance-document/).
Fig. 1

MRMPlus Concept/Flow Diagram. Mass spectrometry data are first preprocessed in Skyline™ and subsequently fed as input to MRMPlus. In addition, MRMPlus takes as inputs, user-defined experiment metadata and a serial dilution information file

Although MRMPlus was conceptualized and implemented to compute the performance of assays developed according to guidelines established by the CPTAC Assay Development Working Group, it may also be used for performance calculations for targeted assays not described by the Working Group.

Implementation

We implemented MRMPlus in the platform-independent Java programing language (Fig. 2). For statistical evaluations, we utilized classes defined in the Apache Commons Mathematics library [23]. For visualizations, we utilized Java classes defined in the JFreeChart library [http://www.jfree.org/jfreechart/].
Fig. 2

MRMPlus’ Graphics User Interface. In addition to user-defined and Skyline input files, MRMPlus’ interface requests other program configuration options

Inputs to the program include Skyline-derived output files in tab-delimited format (Additional files 1 and 4: Tables S1 and S4), a user defined experiment metadata (Additional file 2: Table S2), and a dilution information data file (Additional file 3: Table S3). Current implementation of MRMPlus computes analytical measures for “Response Curve” and “Mini-Validation of Repeatability” experiments according to the ADWG’s current specifications (Additional files 5 and 6: Tables S5 and S6). The response curve serves to identify the lower limit of quantification (LLOQ), the limit of detection (LOD), and the linear range of a peptide of interest in a multipoint serial dilution of a biological replicate or matrix. The mini-validation of repeatability experiments attempt to replicate peptide measurements within a complex mixture over multiple days. Its derived metrics approximate the variability of the assay in real-life practice.

Default runtime initiated interface to MRMPlus accepts inputs to compute “Response Curve” analytical measures and the ‘More Options’ button instantiates the “Mini-Validation of Repeatability” module (Fig. 2). Users may modify the configuration file (MRMPlus.config) in the application package to set runtime default options.

Setting the ‘Visualize Results’ option on the ‘More Options’ panel to ‘TRUE’ instructs MRMPlus to generate performance associated visualizations (Fig. 2). MRMPlus generates 8 visualization panels to provide a global perspective of assay performance across all assayed peptide transitions. In addition, MRMPlus provides peptide-specific (‘Response Curve’ and ‘Repeatability’) performance visualizations. (Additional files 7: Figure S1 and Additional file 8: Figure S2).

Ethics approval

The experimental work that was conducted in this study did not require ethics approval. No patient-derived specimens were used to generate the data for the development of MRMPlus. All of the data were derived from formerly N-linked glycopeptides enriched from commercially-available serum and spiked with crude peptides.

Results and discussions

To demonstrate MRMPlus’ capabilities, we employed it to compute the assay performance metrics on our panel of formerly N-linked glycopeptides. Peptides were selected from in-house discovery phase proteomic experiment analyses of prostate cancer tissues and serum studies. MRM assays to these were developed using commercially available serum and heavy labeled synthetic peptides as spiked-ins. (Please see https://assays.cancer.gov/guidance-document/ for detailed description of experiments). Here in this case study, we subset fifteen of the profiled peptides and their respectively associated transitions for MRMPlus’ demonstration purpose.

The current implementation of MRMPlus outputs results for ADWG-defined experiments’ ‘Response Curve’ and ‘Mini-Validation of Repeatability’. For ‘Response Curve’ estimations, our assay experiments were performed in triplicates over seven dilution ranges (Fig. 3). For ‘Mini-validation of Repeatability’ assay experiments were similarly performed in triplicates, at three dilution (concentration) levels – “High”, “Medium”, and “Low”. These were repeated on five different days (Fig. 4). In both ‘Response Curve’ and ‘Mini-validation of Repeatability’, we assessed three transitions for each peptide.
Fig. 3

MRMPlus‘ Visualization – Response Curve. For each monitored transition of a particular peptide, experimental values are derived at serial dilution levels (calibration points). Experiments at respective dilution level are performed in replicates. Figure shows MRMPlus-generated response curves for a sample peptide – EDALNETR. It includes performance plots for the peptide associated transitions (2.y4.1, 2.y5.1, and 2.y6.1) and summed transition

Fig. 4

MRMPlus‘ Visualization – Mini-validation of Repeatability. For each monitored transition, low, medium and high concentration assays are profiled over five days with replicate experiments on each day. Figure shows MRMPlus-generated ‘repeatability’ performance for a sample peptide – EDALNETR. It includes performance plots for the summed transition at the a low, medium and high concentration level. Y-axis coordinates are median estimate of replicate measurements at the different concentration levels while the X-axis coordinates are the days when the experiments were performed. Current implementation of MRMPlus [by default] generates repeatability performance plot for peptides summed transition

MRMPlus computes performance estimates on two levels for each targeted peptide: 1) each associated transition level and 2) the summation observed transition values. Computed results are presented in tables in which the rows are performance measures for the respectively assayed peptide transition, and the columns are the computed measures of analytical performance. (Additional file 5: Table S5, and Additional file 6: Table S6). It should be noted that no absolute values are reported when the peptide transition values are summed with respect to ‘PVSpec.maxDeviation’, ‘PVSpec.meanAtMaxDev’, ‘PVSpec.dayAtMaxDev’, and ‘PVSpec.concLevelAtMaxDev’ performance measures, because these are specific to transitions.

For ‘Response Curve’ performances, limit of detection across peptide transitions span several magnitudes with minimum LOD reported as 0.000524, and maximum reported as 2293.62 (Additional file 5: Table S5, ‘LOD.value’ column). In all cases, LOD was assessed using spiked-in concentrations (Additional file 5: Table S5, ‘LOD.usedSpiked’ column) as pre-curve blank mass spectrometry runs reported no detectable levels. This is possibly due to the mass spectrometer's sensitivity or resolving power. A lower of limit of quantification (Additional file 5: Table S5, ‘LLOQ.CaliPoint’ column) could not be estimated for peptide transitions YLGNATAIFFLPDEGK (2.b9.1), AGPNGTLFVADAYK (2.y10.1), and INNTHALVSLLQNLNK (3.b6.1). We observed an almost absolutely linear response across all transitions (Additional file 5: Table S5, ‘Curve.Rsquared’ and ‘Curve.SlopeStdErr(%)’ columns) with minimum standard error percent of the slope of the response curve reported as 0.0030 % and maximum as 4.3979 %.

For ‘Mini-Validation of Repeatability’, MRMPlus reports the intra-assay coefficient of variation, inter-assay coefficient of variation, and the total coefficient of variation for each peptide transition at respectively specified low, medium, and high concentration levels. The lowest total CV at the validated lower limit of quantification 3.3918 was found associated with peptide transition YLGNATAIFFLPDEGK (2.y8.1) (Additional file 6: Table S6, ‘ValidatedLLOQ.concLevel’ and ‘totalCV.Me’ columns).

The CPTAC ADWG recommends that for a transition to be deemed specific to a peptide, in all samples above the lower limit of quantification, no transition ratio (ratios of peak areas of different transitions of same peptide) in each sample at all concentrations should deviate >30 % from the mean. MRMPlus reports the maximum deviation percentage observed for each transition; such that a deviation above the specified 30 % indicates non-specificity of such transition to the associated peptide.

Thomas et al. [24] – recently published, provides a complete description of a study of ours demonstrating the utility of MRMPlus.

Comparing MRMPlus to PanoramaWeb

We compared MRMPlus’ results with comparable outputs derived from PanoramaWeb [25] on our data (Additional file 9: Table S7 and Additional file 10: Table S8). MRMPlus and PanoramaWeb derived exactly the same results for estimation of ‘Repeatability’ performance. With respect to ‘Response Curve’ performance estimation MRMPlus’ and PanoramaWeb results varied. These may particularly be due to their differing methods of estimating linearity. However, Both MRMPlus’ and PanoramaWeb report excellent statistical measure of how close their data are fitted to the regression line – indicated by similar R-squared estimation.

Conclusions

MRMPlus facilitates MRM assay development. By aligning its workflow with Skyline, a previously established tool for building, analyzing, and refining targeted mass spectrometry methods and the resulting data, MRMPlus streamlines and simplifies assay performance estimation. Future developments of MRMPlus are anticipated to be integrated to the Skyline open-source tool for an even more streamlined analytical workflow.

Availability and requirements

Project home: https://bitbucket.org/paiyetan/mrmplusgui, and https://bitbucket.org/paiyetan/mrmplusgui/downloads

Operating system: Platform independent

Programming Language: Java

License: The BSD 3-Clause License

Other requirements: Please see documentation here - https://bitbucket.org/paiyetan/mrmplusgui/wiki/Home.

Any restrictions to use by non-academics: None

Declarations

Acknowledgements

The authors acknowledge all the members of the Center for Biomarker Discovery and Translation, Johns Hopkins Medical Institution; and members of the Assay Development Working Group (ADWG) of the National Cancer Institute (NCI) Clinical Proteomics Tumor Analysis Consortium (CPTAC). MRMPlus’ development was supported by the National Institutes of Health, National Cancer Institute, Clinical Proteomic Tumor Analysis Consortium (CPTAC, U24CA160036) ; the Early Detection Research Network (EDRN, U01CA152813); and the National Heart, Lung, and Blood Institute, Programs of Excellence in Glycosciences (PEG, P01HL107153). The authors also acknowledge the help of Jeffrey Whiteaker, Ph.D., of the Fred Hutchinson Cancer Research Center in Seattle, Washington, with deriving comparable performance results on PanoramaWeb.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Department of Pathology, The Johns Hopkins University School of Medicine

References

  1. Washburn MP, Wolters D, Yates JR. Large-scale analysis of the yeast proteome by multidimensional protein identification technology. Nat Biotechnol. 2001;19(3):242.View ArticlePubMedGoogle Scholar
  2. Ünlü M, Morgan ME, Minden JS. Difference gel electrophoresis. A single gel method for detecting changes in protein extracts. Electrophoresis. 1997;18(11):2071–7.View ArticlePubMedGoogle Scholar
  3. Gygi SP, Rist B, Gerber SA, Turecek F, Gelb MH, Aebersold R. Quantitative analysis of complex protein mixtures using isotope-coded affinity tags. Nat Biotechnol. 1999;17(10):994.View ArticlePubMedGoogle Scholar
  4. Ross PL, Huang YN, Marchese JN, Williamson B, Parker K, Hattan S, et al. Multiplexed protein quantitation in saccharomyces cerevisiae using amine-reactive isobaric tagging reagents. Mol Cell Proteomics. 2004;3(12):1154–69.View ArticlePubMedGoogle Scholar
  5. Ong SE, Blagoev B, Kratchmarova I, Kristensen DB, Steen H, Pandey A, et al. Stable isotope labeling by amino acids in cell culture, SILAC, as a simple and accurate approach to expression proteomics. Mol Cell Proteomics. 2002;1(5):376.View ArticlePubMedGoogle Scholar
  6. Asara JM, Christofk HR, Freimark LM, Cantley LC. A label-free quantification method by MS/MS TIC compared to SILAC and spectral counting in a proteomics screen. Proteomics. 2008;8(5):994.View ArticlePubMedGoogle Scholar
  7. Colangelo CM, Chung L, Bruce C, Cheung K. Review of software tools for design and analysis of large scale MRM proteomic datasets. Methods. 2013;61(3):287.View ArticlePubMedPubMed CentralGoogle Scholar
  8. Liebler DC, Zimmerman LJ. Targeted quantitation of proteins by mass spectrometry. Biochemistry (N Y ). 2013;52(22):3797.View ArticleGoogle Scholar
  9. Lange V, Malmström JA, Didion J, King NL, Johansson BP, Schäfer J, et al. Targeted quantitative analysis of streptococcus pyogenes virulence factors by multiple reaction monitoring. Mol Cell Proteomics. 2008;7(8):1489.View ArticlePubMedPubMed CentralGoogle Scholar
  10. Pan S, Aebersold R, Chen R, Rush J, Goodlett DR, McIntosh MW, et al. Mass spectrometry based targeted protein quantification: Methods and applications. J Proteome Res. 2009;8(2):787.View ArticlePubMedPubMed CentralGoogle Scholar
  11. Picotti P, Aebersold R. Selected reaction monitoring-based proteomics: Workflows, potential, pitfalls and future directions. Nat Methods. 2012;9(6):555.View ArticlePubMedGoogle Scholar
  12. Kondrat RW, McClusky GA, Cooks RG. Multiple reaction monitoring in mass spectrometry/mass spectrometry for direct analysis of complex mixtures. Anal Chem. 1978;50(14):2017–21.View ArticleGoogle Scholar
  13. Domon B, Aebersold R. Mass spectrometry and protein analysis. Science (New York, NY). 2006;312(5771):212.View ArticleGoogle Scholar
  14. Shi T, Su D, Liu T, Tang K, Camp DG, Qian WJ, et al. Advancing the sensitivity of selected reaction monitoring‐based targeted quantitative proteomics. Proteomics. 2012;12(8):1074–92.View ArticlePubMedPubMed CentralGoogle Scholar
  15. Lange V, Picotti P, Domon B, Aebersold R. Selected reaction monitoring for quantitative proteomics: A tutorial. Mol Syst Biol. 2008;4:222.View ArticlePubMedPubMed CentralGoogle Scholar
  16. Peterson AC, Russell JD, Bailey DJ, Westphall MS, Coon JJ. Parallel reaction monitoring for high resolution and high mass accuracy quantitative, targeted proteomics. Mol Cell Proteomics. 2012;11(11):1475.View ArticlePubMedPubMed CentralGoogle Scholar
  17. Brusniak MY, Chu CS, Kusebauch U, Sartain MJ, Watts JD, Moritz RL. An assessment of current bioinformatic solutions for analyzing LC-MS data acquired by selected reaction monitoring technology. Proteomics. 2012;12(8):1176.View ArticlePubMedGoogle Scholar
  18. Whiteaker JR, Halusa GN, Hoofnagle AN, Sharma V, MacLean B, Yan P, et al. CPTAC assay portal: A repository of targeted proteomic assays. Nat Methods. 2014;11(7):703.View ArticlePubMedPubMed CentralGoogle Scholar
  19. Mani DR, Abbatiello SE, Carr SA. Statistical characterization of multiple-reaction monitoring mass spectrometry (MRM-MS) assays for quantitative proteomics. BMC Bioinformatics. 2012;13:1.View ArticleGoogle Scholar
  20. Carr SA, Abbatiello SE, Ackermann BL, Borchers C, Domon B, Deutsch EW, et al. Targeted peptide measurements in biology and medicine: Best practices for mass spectrometry-based assay development using a fit-for-purpose approach. Mol Cell Proteomics. 2014;13(3):907.View ArticlePubMedPubMed CentralGoogle Scholar
  21. Cham Mead JA, Bianco L, Bessant C. Free computational resources for designing selected reaction monitoring transitions. Proteomics. 2010;10(6):1106.View ArticlePubMedGoogle Scholar
  22. MacLean B, Tomazela DM, Shulman N, Chambers M, Finney GL, Frewen B, et al. Skyline: An open source document editor for creating and analyzing targeted proteomics experiments. Bioinformatics. 2010;26(7):966.View ArticlePubMedPubMed CentralGoogle Scholar
  23. Math C. The Apache Commons Mathematics Library. 2014.Google Scholar
  24. Thomas SN, Harlan R, Chen J, Aiyetan P, Liu Y, Sokoll LJ, et al. Multiplexed Targeted Mass Spectrometry-Based Assays for the Quantification of N-Linked Glycosite-Containing Peptides in Serum. Anal Chem. 2015;87(21):10830–8. doi:10.1021/acs.analchem.5b02063. Epub 2015 Oct 21. PubMed.View ArticlePubMedPubMed CentralGoogle Scholar
  25. Sharma V, Eckels J, Taylor GK, Shulman NJ, Stergachis AB, Joyner SA, et al. Panorama: a targeted proteomics knowledge base. J Proteome Res. 2014;13(9):4205–10.View ArticlePubMedPubMed CentralGoogle Scholar

Copyright

© Aiyetan et al. 2015

Advertisement