- Software
- Open access
- Published:
cytoviewer: an R/Bioconductor package for interactive visualization and exploration of highly multiplexed imaging data
BMC Bioinformatics volume 25, Article number: 9 (2024)
Abstract
Background
Highly multiplexed imaging enables single-cell-resolved detection of numerous biological molecules in their spatial tissue context. Interactive visualization of multiplexed imaging data is crucial at any step of data analysis to facilitate quality control and the spatial exploration of single cell features. However, tools for interactive visualization of multiplexed imaging data are not available in the statistical programming language R.
Results
Here, we describe cytoviewer, an R/Bioconductor package for interactive visualization and exploration of multi-channel images and segmentation masks. The cytoviewer package supports flexible generation of image composites, allows side-by-side visualization of single channels, and facilitates the spatial visualization of single-cell data in the form of segmentation masks. As such, cytoviewer improves image and segmentation quality control, the visualization of cell phenotyping results and qualitative validation of hypothesis at any step of data analysis. The package operates on standard data classes of the Bioconductor project and therefore integrates with an extensive framework for single-cell and image analysis. The graphical user interface allows intuitive navigation and little coding experience is required to use the package. We showcase the functionality and biological application of cytoviewer by analysis of an imaging mass cytometry dataset acquired from cancer samples.
Conclusions
The cytoviewer package offers a rich set of features for highly multiplexed imaging data visualization in R that seamlessly integrates with the workflow for image and single-cell data analysis.
It can be installed from Bioconductor via https://www.bioconductor.org/packages/release/bioc/html/cytoviewer.html. The development version and further instructions can be found on GitHub at https://github.com/BodenmillerGroup/cytoviewer.
Background
Highly multiplexed tissue imaging allows spatial and single-cell-resolved detection of dozens of biological molecules, including proteins and nucleic acids, in situ. These technologies facilitate an in-depth analysis of complex systems and diseases such as the tumor microenvironment [1,2,3] and type 1 diabetes progression [4]. Imaging-based spatial proteomics methods [5] can be broadly divided into fluorescent cyclic approaches such as cyclic immunofluorescence (t-CyCIF) [6] and co-detection by indexing (CODEX) [7], and one-step mass-tag based approaches that include multiplexed ion beam imaging (MIBI) [8] and imaging mass cytometry (IMC) [9].
To fully leverage the information contained in multiplexed imaging data, the application of computational tools for data analysis is necessary. The main analysis steps, irrespective of the biological question, are (1) image quality control, (2) image pre-processing and segmentation, and (3) single-cell and spatial analysis [10]. Interactive image visualization is essential throughout the entire data analysis workflow and supports key analysis tasks such as segmentation quality control, and hypothesis formulation and verification. With an ever-growing user community [11], this creates the need for easy-to-use image viewers that can visualize all possible results obtained during analysis.
However, popular software, such as histoCAT [12], QuPath [13], and others [14, 15], show little interoperability with programming languages and major frameworks for image and single-cell analysis. The recently developed napari image viewer, which operates in Python, bridges the gap between multiplexed image visualization and data analysis [16], but similar tools that operate in the widely used statistical programming language R have not been developed.
Here, we present cytoviewer, an R/Bioconductor package for interactive multi-channel image and segmentation mask visualization in R. The cytoviewer package builds on the cytomapper R/Bioconductor package [17] and extends its static visualization abilities via an interactive and user-friendly shiny application. It provides interactive visualization strategies in a similar fashion as the iSEE package [18] offers for single-cell data and can be seamlessly harmonized with any step of the data analysis workflow in R. Users can overlay individual images with segmentation masks, visualize cell-specific metadata, and download generated images. The cytoviewer package integrates into the Bioconductor framework [19] for single-cell and image analysis leveraging the image handling and analysis strategies from the EBImage Bioconductor package [20] and building on commonly used Bioconductor classes such as SingleCellExperiment, SpatialExperiment [21, 22], and CytoImageList [17]. We showcase the functionality and biological application of cytoviewer by demonstrating visual exploration of an IMC dataset of cancer patients.
Implementation
The R/Bioconductor cytoviewer package leverages the reactive programming framework of the popular R shiny and shinydashboard packages [23], is cross-platform compatible, and launches an interactive web application.
The graphical user interface (GUI) of the cytoviewer package can be opened directly from R and the function call takes up to five arguments (Fig. 1A). Images must be provided as a CytoImageList object [17] containing one or multiple multi-channel images where each channel represents the pixel intensities of one marker. Segmentation masks in CytoImageList format can be added if desired. Segmentation masks are represented as single-channel images containing integer values for cells or zero for background. Furthermore, SingleCellExperiment [21] or SpatialExperiment [22] class objects can be provided to allow single-cell specific metadata visualization. The full functionality of cytoviewer is leveraged when images, segmentation masks, and a metadata object are provided (Fig. 1A). This allows comprehensive image-level and cell-level visualization, enables image overlays with segmentation masks, and cell-specific metadata visualization.
The cytoviewer GUI has three main parts: body, sidebar, and header (Fig. 1B). The body of cytoviewer features the image viewer. The viewer can switch between image-level visualization, which shows the pixel-level intensities of all selected markers either combined (Composite) or separately (Channels), and cell-level visualization, which displays cell-level information on segmentation masks (Masks) (Additional file 5: Supplementary Notes—S1.1/S1.2). Controls for sample selection, and settings for image and mask visualization are found in the sidebar menu. The header section contains the package version, R session information, a help page, and a drop-down menu for image downloads.
For rapid and publication-quality image downloads, the user specifies a file name, selects the image of interest (Composite, Channels, Mask) and the file format (pdf, png) (Additional file 5: Supplementary Notes—S1.3).
Results
To demonstrate the functionality and potential biological applications of the cytoviewer package, we explored an example IMC dataset from the Integrated iMMUnoprofiling of large adaptive CANcer patient cohort project (immucan.eu) (Additional file 5: Supplementary Notes—S1.4). For IMC, tissue sections are stained with antibodies tagged with isotopically pure rare earth metals, the tissue is laser ablated, and tags are detected by mass spectrometry to produce high-dimensional images [9]. Here, we showcase the different viewing modes of cytoviewer by analyses of images from a breast cancer patient (Patient2_003) and outline the rich biological information content for each step (Fig. 1C, Additional file 3: Supplementary Fig. S1).
Image visualization control is split into basic and advanced control modes. Basic controls support the selection of up to six channels with separate color control settings for each (contrast, brightness, gamma, and channel color). In the example shown here, we visualized expression of Ecad, CD8a, and CD68, which are markers for epithelial and tumor cells, CD8+ T cells, and macrophages, respectively (Fig. 1B, Fig. 1C—Top). This image visualization step supports qualitative cell type identification as well as assessment of signal sensitivity and specificity.
In the advanced image control mode, the user can choose to overlay the displayed images with provided segmentation masks (Fig. 1C—Center). Outline color and thickness can be adjusted by the user. This step supports evaluation of cell segmentation quality, which is essential for downstream biological data interpretation. Moreover, the masks can be outlined by cell-specific metadata from the SingleCellExperiment/SpatialExperiment object. For categorical and continuous metadata entries, the user can choose between discrete colors and continuous color palettes, respectively. By outlining the masks with the cell area and cell type information (e.g., tumor), correct phenotype assignment can be visually confirmed (e.g., tumor cells are Ecad+ and tumor cells have larger areas than other cells).
The user can further decide to display the provided segmentation masks (Fig. 1C—Bottom). Coloring of the masks by cell-specific metadata (categorical and continuous) is possible and can be used for visual assessment of spatial tissue features such as tumor cell areas, tumor patch formation, and tumor cell dissemination, among others.
Using image appearance controls, the user can adjust the scale bar length and include legends or image titles. These features can be used for image-level and cell-level visualization and can aid in interpretation of phenotype co-localization such as infiltration of CD8+ T cells into the tumor core, which is a clinically relevant tissue feature [24] (Fig. 1D). Furthermore, the image filters section controls pixel-wise interpolation (default) and allows applying Gaussian filters on the image-level (Additional file 4: Supplementary Fig. S2).
Discussion
The R/Bioconductor cytoviewer package provides a rich set of visualization features including (1) visualization of composite and single-channel images, (2) outlining of cells on images, (3) setting cell outline colors based on cellular metadata (e.g., cell phenotype), (4) coloring segmentation masks based on cellular metadata (e.g., cell area) and (5) easy access and download of generated plots for publication, among others. It supports key analysis tasks including cell segmentation quality control, cell phenotype identification and confirmation, and hypothesis formulation and verification. As such, it forms a crucial basis for visually validating a number of biological questions such as: are the identified cell phenotypes correctly labelled and are there biases due to incorrect segmentation? Do certain cell phenotypes interact and are they spatially aggregated? Are certain cell phenotypes enriched on images of a certain condition?
The cytoviewer package operates on standard data containers of the Bioconductor project [19] such as SingleCellExperiment/SpatialExperiment [21, 22] and CytoImageList [17] and can thus be readily implemented into existing R pipelines for image and single-cell analysis. The versatile and easy-to-use graphical user interface of cytoviewer further allows accessibility to users with little bioinformatics training and coding experience.
Here, we demonstrated the use of cytoviewer by exploring IMC data (Additional file 1, 2). However, data from other multiplexed imaging technologies including t-CyCIF [6], CODEX [7], or MIBI [8], which produce pixel-level intensities and (optionally) segmentation masks, can be interactively visualized with cytoviewer as long as the input format is appropriate.
We have integrated cytoviewer into our widely adopted data analysis pipeline for multiplexed imaging data [10] (https://bodenmillergroup.github.io/IMCDataAnalysis/) and envision that the package will meet the needs of the fast-growing community of highly multiplexed imaging users [11]. Future developments of the package include an integration with modern imaging file types such as OME-NGFF [25] and spatialData [26] to enable image and single-cell analysis in a programming language agnostic fashion.
Conclusions
The newly developed R/Bioconductor cytoviewer package, written in the statistical programming language R, allows interactive visualization and exploration of multi-channel images and segmentation masks. Alongside the related cytomapper package [17], it builds a well-integrated toolbox for highly multiplexed imaging data visualization that can support all major steps of the image and single-cell analysis workflow in R.
Availability and requirements
-
Project name: cytoviewer
-
Project home page: https://github.com/BodenmillerGroup/cytoviewer
-
Operating system(s): Platform independent
-
Programming language: R
-
Other requirements: R > = 4.0
-
License: None
-
Any restrictions to use by non-academics: None
Availability of data and materials
Abbreviations
- CODEX:
-
Co-detection by indexing
- GUI:
-
Graphical user interface
- IMC:
-
Imaging mass cytometry
- MIBI:
-
Multiplexed ion beam imaging
- OME-NGFF:
-
Open microscopy environment-next generation file format
- t-CyCIF:
-
Tissue-based cyclic immunofluorescence
References
Jackson HW, Fischer JR, Zanotelli VRT, Ali HR, Mechera R, Soysal SD, et al. The single-cell pathology landscape of breast cancer. Nature. 2020;578(7796):615–20.
Hoch T, Schulz D, Eling N, Gómez JM, Levesque MP, Bodenmiller B. Multiplexed imaging mass cytometry of the chemokine milieus in melanoma characterizes features of the response to immunotherapy. Sci Immunol. 2022;7(70):eabk1692. https://doi.org/10.1126/sciimmunol.abk1692.
Risom T, Glass DR, Averbukh I, Liu CC, Baranski A, Kagel A, et al. Transition to invasive breast cancer is associated with progressive changes in the structure and composition of tumor stroma. Cell. 2022;185(2):299-310.e18. https://doi.org/10.1016/j.cell.2021.12.023.
Damond N, Engler S, Zanotelli VRT, Schapiro D, Wasserfall CH, Kusmartseva I, et al. A map of human type 1 diabetes progression by imaging mass cytometry. Cell Metab. 2019;29(3):755–68. https://doi.org/10.1016/j.cmet.2018.11.014.
Moffitt JR, Lundberg E, Heyn H. The emerging landscape of spatial profiling technologies. Nat Rev Genet. 2022;23(12):741–59. https://doi.org/10.1038/s41576-022-00515-3.
Lin JR, Izar B, Wang S, Yapp C, Mei S, Shah PM, et al. Highly multiplexed immunofluorescence imaging of human tissues and tumors using t-CyCIF and conventional optical microscopes. Elife. 2018;7:e31657. https://doi.org/10.7554/eLife.31657.
Goltsev Y, Samusik N, Kennedy-Darling J, Bhate S, Hale M, Vazquez G, et al. Deep profiling of mouse splenic architecture with CODEX multiplexed imaging. Cell. 2018;174(4):968–81. https://doi.org/10.1016/j.cell.2018.07.010.
Angelo M, Bendall SC, Finck R, Hale MB, Hitzman C, Borowsky AD, et al. Multiplexed ion beam imaging of human breast tumors. Nat Med. 2014;20(4):436–42. https://doi.org/10.1038/nm.3488.
Giesen C, Wang HAO, Schapiro D, Zivanovic N, Jacobs A, Hattendorf B, et al. Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat Methods. 2014;11(4):417–22. https://doi.org/10.1038/nmeth.2869.
Windhager J, Zanotelli VR, Schulz D, Meyer L, Daniel M, Bodenmiller B, Eling N. An end-to-end workflow for multiplexed image processing and analysis. Nat Protoc. 2023. https://doi.org/10.1038/s41596-023-00881-0
Hickey JW, Neumann EK, Radtke AJ, Camarillo JM, Beuschel RT, Albanese A, et al. Spatial mapping of protein composition and tissue organization: a primer for multiplexed antibody-based imaging. Nat Methods. 2022;19(3):284–95. https://doi.org/10.1038/s41592-021-01316-y.
Schapiro D, Jackson HW, Raghuraman S, Fischer JR, Zanotelli VRT, Schulz D, et al. histoCAT: analysis of cell phenotypes and interactions in multiplex image cytometry data. Nat Methods. 2017;14(9):873–6. https://doi.org/10.1038/nmeth.4391.
Bankhead P, Loughrey MB, Fernández JA, Dombrowski Y, McArt DG, Dunne PD, et al. QuPath: open source software for digital pathology image analysis. Sci Rep. 2017;7(1):16878. https://doi.org/10.1038/s41598-017-17204-5.
Somarakis A, Van UV, Koning F, Lelieveldt B, Höllt T. ImaCytE: visual exploration of cellular micro-environments for imaging mass cytometry data. IEEE Trans Vis Comput Graph. 2021;27(1):98–110.
Schindelin J, Arganda-Carreras I, Frise E, Kaynig V, Longair M, Pietzsch T, et al. Fiji: an open-source platform for biological-image analysis. Nat Methods. 2012;9(7):676–82. https://doi.org/10.1038/nmeth.2019.
Ahlers J, Althviz Moré D, Amsalem O, Anderson A, Bokota G, Boone P, et al. napari: a multi-dimensional image viewer for Python. 2023; Available from: https://zenodo.org/record/8115575
Eling N, Damond N, Hoch T, Bodenmiller B. cytomapper: an R/Bioconductor package for visualization of highly multiplexed imaging data. Bioinformatics. 2020;36(24):5706–8. https://doi.org/10.1093/bioinformatics/btaa1061.
Rue-Albrecht K, Marini F, Soneson C, Lun ATL. iSEE: interactive summarized experiment explorer [version 1; peer review: 3 approved]. F1000Res. 2018;7(741).
Gentleman RC, Carey VJ, Bates DM, Bolstad B, Dettling M, Dudoit S, et al. Bioconductor: open software development for computational biology and bioinformatics. Genome Biol. 2004;5(10):R80. https://doi.org/10.1186/gb-2004-5-10-r80.
Pau G, Fuchs F, Sklyar O, Boutros M, Huber W. EBImage—an R package for image processing with applications to cellular phenotypes. Bioinformatics. 2010;26(7):979–81. https://doi.org/10.1093/bioinformatics/btq046.
Amezquita RA, Lun ATL, Becht E, Carey VJ, Carpp LN, Geistlinger L, et al. Orchestrating single-cell analysis with Bioconductor. Nat Methods. 2020;17(2):137–45. https://doi.org/10.1038/s41592-019-0654-x.
Righelli D, Weber LM, Crowell HL, Pardo B, Collado-Torres L, Ghazanfar S, et al. Spatial experiment: infrastructure for spatially-resolved transcriptomics data in R using bioconductor. Bioinformatics. 2022;38(11):3128–31. https://doi.org/10.1093/bioinformatics/btac299.
Jia L, Yao W, Jiang Y, Li Y, Wang Z, Li H, et al. Development of interactive biological web applications with R/Shiny. Brief Bioinform. 2022;23(1):bbab415. https://doi.org/10.1093/bib/bbab415.
van der Leun AM, Thommen DS, Schumacher TN. CD8+ T cell states in human cancer: insights from single-cell analysis. Nat Rev Cancer. 2020;20(4):218–32. https://doi.org/10.1038/s41568-019-0235-4.
Moore J, Allan C, Besson S, Burel JM, Diel E, Gault D, et al. OME-NGFF: a next-generation file format for expanding bioimaging data-access strategies. Nat Methods. 2021;18(12):1496–8. https://doi.org/10.1038/s41592-021-01326-w.
Marconato L, Palla G, Yamauchi KA, Virshup I, Heidari E, Treis T, et al. SpatialData: an open and universal data framework for spatial omics. bioRxiv. 2023;2023.05.05.539647. Available from: http://biorxiv.org/content/early/2023/05/08/2023.05.05.539647.abstract
Acknowledgements
We thank the Bodenmiller Lab for the continuous support and discussions over the past year.
Funding
Open access funding provided by Swiss Federal Institute of Technology Zurich N.E. was funded by the European Union’s Horizon 2020 research and innovation program under Marie Sklodowska-Curie Actions Grant Agreement No 892225. B.B. was funded by a SNSF project Grant (#310030_205007: Analysis of breast tumor ecosystem properties for precision medicine approaches), an NIH Grant ([UC4 DK108132]), the CRUK IMAXT Grand Challenge, and the European Research Council (ERC) under the European Union's Horizon 2020 Program under the ERC Grant Agreement No. 866074.
Author information
Authors and Affiliations
Contributions
NE conceived the project. LM designed and created the application with input from NE. LM, NE and BB wrote the manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Additional file 1: Publication analysis code.
Analysis code to reproduce present study.
Additional file 2: Video demonstration. A video demonstrating the functionality of cytoviewer.
Additional file 3: Fig. S1.
cytoviewer graphical user interface overview. The graphical user interface of cytoviewer for the three different viewer modes. Image-level-Composite with basic controls (Top-Left) and advanced controls (Top-Right), Image-level-Channels with basic controls (Middle-Left) and advanced controls (Middle-Right) and Cell-level-Mask with basic controls (Bottom-Left) are shown. For image-level visualization, Ecad (magenta), CD8a (cyan) and CD68 (yellow) marking tumor cells, CD8+ T cells and myeloid cells, respectively, are shown and channel color settings are as follows for all markers: Contrast: 2,5; Brightness: 1; Gamma: 1.2. For cell-level visualization, tumor cells (magenta) are highlighted. Note that the Composite and Mask tabs are zoomable. Scale bars: 150 µm.
Additional file 4: Fig. S2.
cytoviewer image filters. Image filter controls are relevant for the image level (here: Composite). Ecad (magenta), CD8a (cyan) and CD68 (yellow) marking tumor cells, CD8+ T cells, and myeloid cells, respectively, are shown. Channel color settings are as follows for all markers: Contrast: 2,5; Brightness: 1; Gamma: 1.2. The user can turn on pixel-wise interpolation (Left, default) and off (Center). The white boxes indicate the areas magnified in lower images. Users can also apply a Gaussian filter to the image (Right, sigma: 1.5). Scale bars: 150 µm.
Additional file 5:
Supplementary Notes.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Meyer, L., Eling, N. & Bodenmiller, B. cytoviewer: an R/Bioconductor package for interactive visualization and exploration of highly multiplexed imaging data. BMC Bioinformatics 25, 9 (2024). https://doi.org/10.1186/s12859-023-05546-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12859-023-05546-z