A biosegmentation benchmark for evaluation of bioimage analysis methods
© Drelie Gelasca et al; licensee BioMed Central Ltd. 2009
Received: 26 November 2008
Accepted: 01 November 2009
Published: 01 November 2009
We present a biosegmentation benchmark that includes infrastructure, datasets with associated ground truth, and validation methods for biological image analysis. The primary motivation for creating this resource comes from the fact that it is very difficult, if not impossible, for an end-user to choose from a wide range of segmentation methods available in the literature for a particular bioimaging problem. No single algorithm is likely to be equally effective on diverse set of images and each method has its own strengths and limitations. We hope that our benchmark resource would be of considerable help to both the bioimaging researchers looking for novel image processing methods and image processing researchers exploring application of their methods to biology.
Our benchmark consists of different classes of images and ground truth data, ranging in scale from subcellular, cellular to tissue level, each of which pose their own set of challenges to image analysis. The associated ground truth data can be used to evaluate the effectiveness of different methods, to improve methods and to compare results. Standard evaluation methods and some analysis tools are integrated into a database framework that is available online at http://bioimage.ucsb.edu/biosegmentation/.
This online benchmark will facilitate integration and comparison of image analysis methods for bioimages. While the primary focus is on biological images, we believe that the dataset and infrastructure will be of interest to researchers and developers working with biological image analysis, image segmentation and object tracking in general.
Quantitative measures derived from microscopy images are basic to enhancing our understanding of biological processes. With the rapid growth in emerging imaging technologies and high throughput bioimaging, robust image processing methods are critically needed in such quantitative analysis. While there is a large amount of literature concerning basic image processing methods, there exists currently no proper guidance for an end-user to choose a small subset of methods that are likely to be effective in a given application scenario. This is particularly true for segmentation and tracking, where literally hundreds of new methods are proposed each year. In most of these cases experimental results are provided on a very limited set of data, often coming from different domains, making it more difficult to judge their usability. The lack of well defined data sets that allow a fair comparison of different basic methods is a major bottleneck for progress in bioimage analysis. This is the main motivation in building the biosegmentation benchmark infrastructure and dataset collection for biological image analysis applications. In particular, we have collected datasets of different modalities and scales and carefully generated manual ground truth that could be of significant help not only to researchers in biological image analysis but also to the image processing community in general. By having a standardized set of data with associated ground truth, we believe that rapid progress can be made not only in identifying the appropriate methods for a particular task but also in facilitating the development of new and more robust methods.
In this paper we focus specifically on a benchmark dataset for image segmentation and tracking. Typical challenges in developing robust bioimage analysis methods include low signal to noise ratio, complex changes in object morphology and the diversity of imaging techniques (such as confocal, bright-field, electron microscopy, phase contrast imaging). Given this diversity in imaging methods and bioimage samples, it is now well recognized that there is a clear need for validating new image analysis methods, see for example [1, 2].
Benchmarks can be invaluable tools for both image processing specialists and scientists. The developers of the algorithms can use such benchmarks to evaluate the performance, reliability and accuracy of newly developed methods. The benchmark provides them with a well established problem set. Further, the workload involved in validation can be reduced significantly by providing access to other analysis and evaluation methods .
There have been several successful benchmarking efforts in image analysis and computer vision, such as the face recognition dataset , the Berkeley (University of California Berkeley) segmentation dataset for natural images  and the object Caltech (California Institute of Technology) 101 dataset . In medicine, databases with macrobiological structures such as mammogram and Magnetic Resonance images , and clinical data  have also been developed. In biology, there have been some efforts in creating microbiological image databases such as the Cell Centered Database  and the Mouse Retina Database . The Protein Classification Benchmark Collection  was created in order to collect a standard datasets on which the performance of machine learning methods can be compared. Finally, the Broad Bioimage Benchmark Collection  consists of microscopy image sets with associated ground truth, such as cell counts, foreground/background and object outlines.
In addition to the above datasets, there have been few organized competitions in computer vision. These include the Face Recognition Grand Challenge (FRGC) , Face Recognition Vendor Test (FRVT) 2006 , and the Iris Challenge Evaluation 2006 . Data and evaluation results of Iris Recognition competition are available in  and Benchmarking Change Analysis Algorithms in Lung CT in . Results from FRGC and FRVT 2006 challenges documented two orders of magnitude improvement in the performance of face recognition under full-frontal, controlled conditions over the last 14 years. Similarly, researchers have reported a significant improvement in object recognition performance over the Caltech 101 and Caltech 256 datasets over the last few years. This further supports our earlier observation that good benchmark datasets with ground truth information can act as catalysts in the development of robust image analysis methods.
A preliminary version of this dataset was presented in a conference publication at the International Conference Image Processing'08 . This work expands on  by providing detailed descriptions on segmentation algorithms and performance metrics. In addition a new 3D image dataset is included. Also, we describe a web-accessible infrastructure that we have developed recently for testing the algorithms. A flexible metadata model (see Section Availability and Requirements and Appendix) is described that is used to exchange data and results for performance evaluation. This infrastructure lowers the burden of choosing datasets for testing algorithms, re-implementing analysis methods and developing evaluation metrics for comparison.
Results and Discussion
Our biosegmentation benchmark  consists of:
image datasets at different scales;
ground truth, manually verified results (e.g. segmentation, cell counting, tracking data);
analysis methods, mostly segmentation methods, cell counting and tracking algorithms;
evaluation methods, image analysis performance measurement;
Subcellular level. Datasets and ground truth in the benchmark at subcellular level.
512 × 600 × 30
hamster, human (HUVEC)
1374 traces of microtubules
Cellular level. Datasets and ground truth in the benchmark at cellular level.
Cat retinal photoreceptors
Arabidopsis and cat retinal 3D cells
Breast cancer cells
COS1 kidney cells
512 × 512 (also 768 × 512)
512 × 521 × 50 up to 1056 × 1056 × 30
896 × 768 (also 768 × 512)
1024 × 1024
nuclear and membrane stains
H & E stain
Calcein AM (green-alive) Propidium iodide (red-dead) Hoechst (nuclear stain-blue)
2, 6, 12,
cell count and centroid
58 binary masks
5 binary masks
Tissue level. Datasets and ground truth in the benchmark at tissue level.
Retina layer detection
300 × 200
Rod photoreceptors (α Rod opsin) Muller cells (α GFAP) Microglia (isolectin B4)
normal, 1-day, 3-day 7-day, 28-day detached
91 layer masks 108 boundary masks
Example implementations of image analysis tools are included for comparing newly developed algorithms. These include detection and tracking at subcellular level; cell counting and segmentation to quantify cellular structures at cellular level and layer segmentation at the tissue level. Researchers can compare the performance of their algorithms through established evaluation metrics such as precision and recall measures. Furthermore, scientists can use this benchmark as a resource for finding the best analysis methods available. All these data and tools are available through a web accessible infrastructure .
At the subcellular level, the structures within a cell have a typical size of less than 1 μ m. Cells consist of organelles that are adapted and/or specialized for carrying out one or more vital functions and large assemblies of macromolecules that carry out particular and specialized functions. Such cell structures, which are not formally organelles, include microtubules. Our example dataset at subcellular level consists of time sequence images of microtubules under different experimental conditions. The image analysis challenges at this scale and with the fluorescence imaging acquisition method are typical for in-vivo subcellular imaging in that the analysis methods need to cope with high clutter and low signal to noise ratio.
The microtubule dataset (Table 1) is obtained by transmitted light microscopy at the Feinstein/Wilson Laboratory at University of California, Santa Barbara (UCSB). The microtubule dataset includes 1374 traces which consists of ground truth for both microtubule tip location and microtubule bodies.
The manual measurements of these microtubules are very labor intensive and time consuming. To obtain an automatic quantitative description of behavior under different experimental conditions, tracing algorithms have been implemented. Due to the limitations in biological sample preparation and inconsistent staining, typical images in live cell studies are noisy and cluttered, making automatic microtubule tracing difficult. Our benchmark implementation includes an automatic method that employs arc-emission Hidden Markov Model for extracting curvilinear structures from live cell fluorescence images .
microtubule tip distance, ϵ t : tip distance error is the Euclidean distance between the ground truth tip to the analysis trace tip,
microtubule trace body distance, ϵ d : trace distance error is the average distance from all the points on the ground truth to all the points on the trace,
microtubule length errors, ϵ l : length difference is simply the difference between the length of the ground truth and the trace.
where the thresholds τ t , τ b and τ l , are empirically set by biologists.
The cell is the structural and functional unit of all known living organisms. A typical cell size is 10 μ m. Image processing challenges at the cellular level include large variations in cell phenotype, staining intensity variation within and across the images, and occlusions. Cells can grow, reproduce, move and die during many processes, such as wound healing, the immune response and cancer metastasis. One of the common tasks is to count the number of cells or nuclei, particularly in histological sections, and characterize various cell attributes, such as cell density, morphology, size and smoothness of the boundary. In our example datasets, we use cell counting as a feature for estimating cell density in 2D retinal and 3D Arabidopsis images, and cell segmentation for studying cell morphology in breast cancer and kidney histopathological images.
Photoreceptors in Retinal Images
2D cell nuclei detection
As mentioned above, of particular interest to this collection is detection of cell nuclei. We have implemented in the benchmark system a 2D nuclei detector based on a Laplacian of Gaussian blob detector, see  for more details on the method itself.
Common ways of evaluating cell/nuclei counting take into account the mismatched counts between detected and ground truth nuclei, and/or the displacement of detected nuclei. In 2D analysis evaluation only the counts are available in the ground truth.
where N is the number of images in the dataset, ND i and are the number of nuclei automatically detected and the average of manual counting, respectively.
Our nuclei detector  applied to the 2D retina dataset gives an error of 3.52% for the nuclei count within the ONL retina layer.
Cell Nuclei in 3D Plant Images (Arabidopsis)
In plants, meristems are regions of cells capable of division and growth. The live 3D imaging of the Arabidopsis meristem has been recently applied in order to analyze the cell lineage and the cell fate during active growth of the shoot meristem . This technique helps to understand the genetic control of the meristem size. Again, cell counts are often used to quantify this process. However, this is an extremely time consuming and laborious task given that a 3D stack consists of approximately 1700 cells.
3D cell nuclei detection
The 3D nuclei detection  extends our earlier work on 2D detection based on a Laplacian of Gaussian blob detector and it is also integrated into our benchmark. Figure 7(b) shows Arabidopsis's nuclei automatically detected in the selected region of the image shown in Figure 7(a).
where false positives (f p ) are objects detected in the test image but not present in the ground truth, false negatives (f n ) are objects that were not detected in the test image but are present in the ground truth. GT and ND denote, respectively, the ground truth (human computed) and automatically detected nuclei coumt. The mean distance ( ) is the mean of all the distances between the detected nuclei locations and their corresponding ground truth locations, (σ d ) is the standard deviation of these distances, and d max is the max of all the distances. Note that G3Dis normalized to [0, 1] and 1 represents the worst case. To quantify the performance of our automatic 3D nuclei detection algorithm , we compared its output with ground truth manually annotated by two experts and obtained a G3Dof 0.1605. When we compared one expert ground truth to the other one we obtained a G3Dof 0.1260.
Breast Cancer Cells
The utility of determining nuclear features for correct cancer diagnosis has been well established in medical studies. Scientists extract a variety of features from nuclei in histopathology imagery of cancer specimens, including size, shape, radiometric properties, texture, and chromatin-specific features. Histopathological images are stained since most cells are essentially transparent, with little or no intrinsic pigment. Certain special stains, which bind selectively to particular components, are used to identify biological structures such as cells. Routine histology uses the stain combination of hematoxylin and eosin, commonly referred to as H&E. In those images, the first step is manual cell segmentation for subsequent classification into benign and malignant cells.
In our benchmark dataset there are 58 H&E stained histopathology images used in breast cancer cell detection from David Rimm's Laboratory at Yale. The ground truth is obtained for 50 images including both benign and malignant cells and is described in Table 2.
Breast cancer cell segmentation
where N is the number of ground truth nuclei; N D is the number of nuclei detected by the segmentation algorithm; the weight α1 can be thought of as the penalty for an over-segmented nucleus; SR is the number of segmented regions overlapping the ground truth nucleus; δ SR = 1 is the upper limit for number of segmented regions; PM is the number of pixels missing; GT is the number of pixels in the ground truth; QS PM is the "quadrant sum" of the pixels missed; α2 can be thought of as the penalty regions of pixels missed, penalizing both size and shape; EP the number of excess pixel; α3 is thus the penalty for size and shape of excess pixel regions, and is related to the degree of under-segmentation of the nucleus; QS EP is "quadrant sum" of the excess pixels; the term with α4 = 1 is simply the detection rate; ER as the number of excess segmented regions and δ ER is the fraction of total ground truth nuclei that we will allow as excess regions; α5 is the penalty for excess segmented regions. The metric takes value P ⊂ [0, 1] and 1 represents the worst segmentation scenario. For a detailed explanation of the metric the reader is referred to [28, 29]. Our cell segmenter using the seeds from the 2D cell nuclei segmentation gets a score of P = 0.25.
In addition to the above dataset we also have images of kidney cells and ground truth corresponding to kidney cell segmentation, but we do not have any associated analysis or evaluation methods for this data. This data was collected by Feinsten's Lab at UCSB to study Alzheimer's disease. Usually manual segmentation provides a reliable alive/dead cell ratio which will test the hypothesis that tau (which is a protein) confers an acute hypersensitivity of microtubules to soluble, oligomeric amyloid-beta and that Taxol, a microtubule-stabilizing drug, provides neuroprotective effects. Because tau is not endogenously expressed, tau effects are easier to study in kidney cancer cells. Kidney cells can easily be transfected with tau. To quantify this phenomenon, COS1 cells (immortalized African monkey kidney cells) are collected through confocal microscopy imaging at the Feinsten's Lab at UCSB.
In the dataset (see Table 2) the images are of both wild-type COS1 cells (non-transfected) and tau transfected COS1 cells and these cells are imaged at 7 different time points after treatment (2 hrs, 6 hrs, 12 hrs, 48 hrs, 72, hrs, and 120 hrs). Ground truth has also been collected for 5 images of this dataset and is represented by binary masks.
Retinal layer segmentation
This modified F measure allows for weighting more segmentation errors in larger layers.
Our best performing method  gives a F-measure around 88% when applied on the dataset. The average distance between boundary pixels in the computed and ground truth data using the boundary detection method from , averaged over all experimental conditions, is 9.52 pixels.
Benchmark datasets often have a strong positive effect in advancing the state of the art of image analysis algorithms.
The benefits of benchmarking include a stronger consensus on the research goals, collaboration between laboratories, transparency of research results, and rapid technical progress. Our benchmark provides unique, publicly available datasets as well as image analysis tools and evaluation methods. The benchmark infrastructure avoids the burden of choosing datasets for testing algorithms, reimplementing analysis methods and evaluation metrics for comparison.
We hope that our benchmark will help researchers to validate, test and improve their algorithms, as well as provide biologists a guidance of algorithms' limitations and capabilities. The benchmark datasets and methods are available online . Analysis results can be uploaded directly and automatically evaluated. The benchmark data that we describe is by no means complete, given the complexity and diversity of the bio-samples and imaging modalities. By making this infrastructure easily accessible to the community, we hope that the collections and analysis methids will grow over time. Users are encouraged to submit datasets, associated ground truth, and analysis results for evaluation. Moreover, user contributed analysis (e.g. segmentation) algorithms and evaluation methods will be integrated upon request.
Availability and Requirements
It is generally acknowledged that making fair comparisons between different methods is quite difficult, particularly in the absence of well established datasets. In addition, even if such datasets are available, often the researchers are left to implement other methods on their own in order to make such comparisons. Newly developed algorithms are often tested against relatively limited datasets. Keeping these limitations in mind, we have been developing the UCSB Bisque infrastructure [18, 33] whose primary goal is to facilitate a tighter integration of datasets and analysis methods.
In the following, we outline the Bisque  components that are relevant to the benchmarking effort. All of the working modules and datasets are available from our bioimage informatics website . From the website users can download the different datasets discussed in this paper and associated ground truth data. Each dataset includes a complete set of original images to process, a document in XML format and an example of ground truth. The xml structure follows Bisque standards  and examples for metadata and graphical annotations are presented in the Appendix. A complete description and formatting of metadata for each dataset is given in .
Two evaluation options are available to the users (the evaluation procedures implement the scoring methods discussed in the previous sections for the different datasets):
Matlab code for evaluation is available for download. Users can download this code and self evaluate the performance of the analysis methods. The evaluation requires that the results are stored in a certain format and a detailed description of the file formats are also provided on Bisque,
users can also evaluate the performance using our web-based evaluation module. In order to use the web-based evaluation the users must first register into Bisque. They need to first upload their results, one image at the time in the correct format to the web-based evaluator and the evaluation results will be automatically displayed on the web site. This option will also allow the result to be stored on our benchmark website and made available to the registered users.
Flexible data model: The benchmark uses a flexible metadata model based on tag documents to store and process ground truth and resultant data. A tag is a named field with an associated value. The tags themselves may be nested, and include values that are strings, numbers, other documents or list. For example, the 3D cell counting document has:
<benchmark type="3D nuclei"> <image name="1. tiff " src ="1. tiff ">
<gobject name="my_algorithm " type="NucleiDetector3D_automatic">
<gobject name="1" type="point">
<vertex x="0" y="1020" z="0"/>
<gobject name="2" type="point">
<vertex x="1048" y="941" z="4"/>
<gobject name="3" type="point">
<vertex x="871" y="1046" z="2"/>
The benchmark uses also an extensible metadata format for graphical annotations that has a number of graphical primitives and can be extended by new object types and object properties. Graphical annotations are termed gobject s and are used, to represent ground truth objects in the dataset. The following is an example gobject description for microtubules:
<gobject name="gt " type="mt_gt" >
<tag name="expert " value="expert_name"/>
<tag name="tube_id " value ="1"/>
<polyline type="polyline " name="polyline">
<vertex x="235.503009" y="170.699054" index="0" t ="0.000000"/>
<vertex x="246.143594" y="174.614789" index="1" t ="0.000000"/>
<polyline type="polylin e " name="polyline">
<vertex x="235.503009" y="170.699054" index="0" t ="1.000000"/>
<vertex x="250.144454" y="175.891660" index="1" t ="1.000000"/>
- Caltech :
California Institute of Technology
- COS :
CV-1 (simian) in Origin, and carrying the SV40 genetic material
- CT :
- FRGC :
Face Recognition Grand Challenge
- FRVT :
Face Recognition Vendor Test
- GCL :
Ganglion Cell Layer
- H&E :
- INL :
Inner Nuclear Layer
- IS :
- MRI :
Magnetic Resonance Imaging
- ML :
- ONL :
Outer Nuclear Layer
- OPL :
Outer Photoreceptor Layer
- OS :
- UCSB :
University of California Santa Barbara.
This research is supported by NSF ITR-0331697 and NSF III-0808772. The authors would like to thank Emre Sargin, Jiyun Byun, Luca Bertelli, Nhat Vu, Pratim Ghosh, Laura Boucheron, Geoff Lewis, DeeAnn Hartung, Brian Ruttenberg, Matt Strader, Jose Freire, Sharath Venkatesha and Santhoshkumar Sunderrajan for their valuable help in developing the benchmark and Camil Caizon, Eden Haven, Stephanie Perez, Corey Cox and Nicholas Secchini Drelie for providing ground truth.
The bioimage retinal datasets are contributed by the Steven Fisher's Lab. (Retinal Cell Biology Lab, Department of Molecular, Cellular and Developmental Biology, UCSB), the microtubule and kidney cells image by Stuart Feinstein and Leslie Wilson's Lab. (Neuroscience Research Institute, Department of Molecular, Cellular and Developmental Biology, UCSB), the 3D cells image by Elliot Meyerowitz and Marcus Heisler (Division of Biology, California Institute of Technology) and the breast cancer dataset by David Rimm (Department of Pathology, Yale University School of Medicine).
- Jannin P, Krupinski E, Warfield S: Guest editorial validation in medical image processing. IEEE Transactions on Medical Imaging 2006, 25(11):1405–1409.View ArticlePubMedGoogle Scholar
- Tadeusiewicz R, Ogiela MR: Medical image understanding technology, it Volume 156 of Studies in Fuzziness and Soft Computing. Berlin-Heidelberg-New York: Springer-Verlag; 2004.Google Scholar
- Phillips PJ, Moon H, Rauss PJ, Rizvi S: The FERET evaluation methodology for face recognition algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 2000., 22(10):Google Scholar
- Martin D, Fowlkes C, Tal D, Malik J: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. International Conference on Computer Vision, Vancouver, BC, Canada 2001, 2: 416–423.Google Scholar
- Fei-Fei L, Fergus R, Perona P: One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence 2006, 28(4):594.View ArticlePubMedGoogle Scholar
- Nattkemper TW, Twellmann T, Schubert W, Ritter H: Human vs. machine: Evaluation of uorescence micrographs. Computers in Biology and Medicine 2003, 33: 31–43.View ArticlePubMedGoogle Scholar
- NIBIB's image and clinical data2008. [http://www.nibib.nih.gov/Research/Resources/ImageClinData]
- The cell centered database2002. [http://www.ccdb.ucsd.edu]
- Mouse retina SAGE library2004. [http://cepko.med.harvard.edu/]
- Protein classification benchmark collection2006. [http://net.icgeb.org/benchmark]
- Broad bioimage benchmark collection2008. [http://www.broad.mit.edu/bbbc]
- Face recognition grand challenge2008. [http://face.nist.gov/frgc]
- Face recognition vendor test2006. [http://face.nist.gov/frvt]
- Iris challenge evaluation[http://iris.nist.gov/ice]
- FRVT 2006 and ICE 2006 large-scale results In Tech. Rep. NISTIR 7408. National Institute of Standards and Technology; 2007.
- Gelasca ED, Byun J, Obara B, Manjunath BS: Evaluation and benchmark for biological image segmentation. IEEE International Conference on Image Processing, San Diego, CA, USA 2008, 1816–1819.Google Scholar
- Kvilekval K, Fedorov D, Obara B, Manjunath BS, Singh AK: Bisque: bio-image semantic query user environment.2008. [http://bioimage.ucsb.edu/bisque]Google Scholar
- UCSB biosegmentation benchmark2008. [http://bioimage.ucsb.edu/biosegmentation]
- Jordan MA, Wilson L: Microtubules as a target for anticancer drugs. Nature Reviews Cancer 2004, 4: 253–265.View ArticlePubMedGoogle Scholar
- Altinok A, El-Saban M, Peck A, Wilson L, Feinstein S, Manjunath BS, Rose K: Activity analysis in microtubule videos by mixture of hidden Markov models. IEEE Computer Society Conference on on Computer Vision and Pattern Recognition, New York, NY, USA 2006., 2:Google Scholar
- Panda D, Samuel JC, MMassie , Feinstein SC, Wilson L: Differential regulation of microtubule dynamics by three- and four-repeat tau: Implications for normal neuronal development and the onset of neurodegenerative disease. Proceedings of the National Academy of Sciences of the USA 2003, 100(16):9548–9553.PubMed CentralView ArticlePubMedGoogle Scholar
- Sargin ME, Altinok A, Rose K, Manjunath BS: Deformable trellis: Open contour tracking in bio-image sequences. IEEE International Conference on Acoustic, Speech and Signal Processing, Las Vegas, NV, USA 2008, 561–564.Google Scholar
- Eibl KH, Lewis GP, Betts K, Linberg KA, Gandorfer A, Kampik A, Fisher SK: The effect of alkylphosphocholines on intraretinal proliferation initiated by experimental retinal detachment. Investigative Ophthalmology and Visual Science 2007, 48(3):1305–1311.View ArticlePubMedGoogle Scholar
- Byun J, Verardo MR, Sumengen B, Lewis GP, Manjunath BS, Fisher SK: Automated tool for the detection of cell nuclei in digital microscopic images: Application to retinal images. Molecular Vision 2006, 12: 949–960.PubMedGoogle Scholar
- Gordon SP, Heisler MG, Reddy GV, Ohno C, Das P, Meyerowitz EM: Pattern formation during de novo assembly of the Arabidopsis shoot meristem. Development 2007, 134(19):3539–3548.View ArticlePubMedGoogle Scholar
- Obara B, Byun J, Fedorov D, Manjunath BS: Automatic nuclei detection and data flow in Bisquik system. Workshop on Bio-Image Informatics: Biological Imaging, Computer Vision and Data Mining, Santa Barbara, CA, USA 2008.Google Scholar
- Boucheron L: Object- and spatial-level quantitative analysis of multispectral histopathology images for detection and characterization of cancer. PhD thesis. University of Santa Barbara, CA, USA; 2007.Google Scholar
- Boucheron LE, Harvey NR, Manjunath BS: A quantitative object-level metric for segmentation performance and its application to cell nuclei. International Symposium on Visual Computing, Lake Tahoe, NV, USA 2007, 4841: 208–219.Google Scholar
- Bertelli L, Byun J, Manjunath BS: A variational approach to exploit prior information in object-background segregation: Application to retinal images. IEEE International Conference on Image Processing, San Antonio, TX, USA 2007., 6:Google Scholar
- Vu N, Ghosh P, Manjunath BS: Retina layer segmentation and spatial alignment of antibody expression levels. IEEE International Conference on Image Processing, San Antonio, TX, USA 2007., 2:Google Scholar
- Bertelli L, Gosh P, Manjunath BS, Gibou F: Reference-based probabilistic segmentation as non-rigid registration using thin plate splines. IEEE International Conference on Image Processing, San Diego, CA, USA 2008.Google Scholar
- Kvilekval K, Fedorov D, Obara B, Ruttenberg B, Manjunath BS, Singh AK: Bisque: bio-image semantic query user environment. Workshop on Bio-Image Informatics: Biological Imaging, Computer Vision and Data Mining, Santa Barbara, CA, USA 2008.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.