BIOCAT: a pattern recognition platform for customizable biological image classification and annotation

Design of BIOCAT

BIOCAT is designed with 1) usability, 2) extensibility, 3) inter-operability, and 4) portability. Figure 1 shows some screenshots of the BIOCAT GUI.

Modularized algorithms and algorithm chain

The BIOCAT’s algorithms are modularized to facilitate the adaptive comparison of what we call “algorithm chains”, which is a sequence of several pattern recognition algorithms that include some feature extractors, one or more optional feature selector(s), and a classifier.

For feature extraction, we consider 2D and 3D textural, morphological and structural features. Texture features are an important category of effective image features such as wavelet features. Morphological features describe the shape and structure of 2D and 3D objects. A pixel in a 2D digital image consists of 4 or 8 neighbor pixels depending on whether the diagonal neighbors are considered. Some 3D features can be extended from 2D by extending an 8-connected neighborhood to a 3D 26-connected neighborhood. Examples of morphological features include Hu moments, which are statistical summaries of intensity in the 2D or 3D object neighborhood, and measures in relation to the center of mass of the images [28] or Zernike moments which are orthogonal invariant moments using a set of complex polynomials [29]. Other algorithms include Hessian features which are suitable for detecting tube-like neuronal structures, Gaussian derivatives and Laplacian features [30], as well as object statistics [21]. Currently 14 different feature extractors are available in BIOCAT with the following examples, each providing one to multiple measures extracted from an image:

BIOCAT has feature selectors such as Fisher’s criterion that select a subset of features. BIOCAT also provides several classifiers including support vector classifiers [31], nearest neighbor, Naïve Bayes, decision tree, and random forest.

As explained in Implementation Section, BIOCAT supports an extensible design, so the above algorithms are of a growing list. Currently, BIOCAT has about 20 algorithms for 2D/3D feature extractors, selectors and classifiers. The list includes the 3D anisotropic wavelet feature extractor and re-implementations of known algorithms (a majority of the feature extractors are re-implementations). The (re)implementations are memory- and efficiency-aware for working with high content multidimensional bioimages. These feature extractors in BIOCAT reuse the memory space for each image. In addition, due to the extensible design, libraries may be wrapped and incorporated as plugins whenever such algorithms become available. For example, Hessian features is a plugin wrapped around FeatureJ implementation [30].

Note that the “algorithm chain” in our paper refers to a sequential flow of feature extraction, selection and classification, which is different from the narrower sense of supervised (or unsupervised) learning of pattern recognition. When multiple feature extractors are used, these features are linearly aggregated. The combined feature’s dimensionality is the sum of all individual feature vectors. Such combination may lose some property (e.g. invariance) of certain features if they are not combined with features with similar property, yet it provides the flexibility for extending to many feature extractors that are often useful for biological image classification. When multiple selectors are used, it is a cascading design: the feature vector goes through the first selector. The resulting feature vector is then filtered by the second selector and so on.

With all the algorithm modules as building blocks, BIOCAT can conveniently build algorithm chains to compare multiple alternative models. One example algorithm chain can be an anisotropic 3D wavelet feature extractor along with a 3D Hu Moments feature extractor, followed by a Fisher feature selection and ultimately a support vector classifier. The chains with the same types of algorithms but different parameters are considered to be different chains, so that dissimilar parameters (e.g. multiple wavelets) can be compared to find the suitable ones for the model. Figure 1b shows BIOCAT’s screenshot of building and comparing the algorithm chains.

3D anisotropic wavelet texture feature

One important category of effective image features is texture features, which is a family of features that measure the texture of images such as wavelet features. In particular, wavelet features are obtained using discrete wavelet transform (DTW) with wavelet functions [32]. Two dimensional wavelet features had performed well in our previous work with gene-expression annotation of fruit fly [16]. However, full extension of these features from 2D to 3D can lead to a big increase in the number of features, and consequently the storage and computational need. For example, for wavelet transform, a full extension to 3D wavelet features will lead to a number of features that is cubic to the side length of the object (instead of being quadratic in 2D). We thus designed a multi-scale anisotropic 3D wavelet feature. Such features can be particularly suitable for 3D confocal microscopic images since it adapts to the anisotropic nature of confocal imaging where z resolution is typically less than x-y resolution.

where I is the intensity, the size of the extracted volume is (2*rx+1)*(2*ry+1)*(2*rz+1), ψ is the discrete Haar wavelet basis function, k, n are the dilation and translation factors of wavelet with k=0,1 (for two-level wavelets), n=0....(2rx+1)*(2ry+1), and (2*rz+1) is the number of slices included in calculation. w(.) is a Gaussian function with middle z slices weighted heavier than other slices.

The multi-scale component of the 3D anisotropic wavelet is represented by the dilation factor k of the wavelet transform. The number of the scales is also called wavelet level. At each level, a Haar-based discrete wavelet transform is done on the approximation image from the previous finer resolution. Figure 2 shows an example of level-2 3D isotropic wavelet, where 2*rz+1 slices are involved in calculating the feature for a 3D image of a fluorescently labeled fruitfly brain.

Note that other tools also use wavelet for image analysis. For example, Icy [11]’s spot detector plugin [33] uses wavelet adaptive threshold for object detection, which is an unsupervised scheme different from the method presented in BIOCAT.

2D/3D image set classification and model selection

Table 2 also gives the comparison of performances between BIOCAT and the benchmarking results in literature. Literature results are from the Wndchrm tool which achieves the best accuracy on the sets so far. Wndchrm has a fixed option of extracting about 1000 or 2600 features regardless of the problem. For comparison, we also report the number of features used by BIOCAT in Table 2.

Results in Table 2 show that BIOCAT can effectively classify various biological image sets. For nine out of the thirteen data sets, BIOCAT achieves the state-of-the-art or better accuracy. For Binuleate and LiverGenderCR, only 40 features per image were used to achieve about 100% accuracy. For MuscleAge, LiverAging and LiverGenerAL sets, BIOCAT is able to greatly improve the current best results. For the CHO set, by using object statistics alone, which is a set of 7 features, BIOCAT achieves comparable results with Wndcharm and 6.3% more than the original literature (Random Forest is used as the classifier). Comparatively, Wndchrm needs a much larger number of features to achieve the reported results. Wndchrm’s need of calculating thousands of complex features can lead to a much higher computational complexity which would make it slow on larger images (Wndchrm does not have a classification functionality for 3D images which would be more computational demanding.) BIOCAT’s adaptive design, on the other hand, allows the best suitable algorithms to be selected for a given biological image classification problem, which can sometimes be simple, as in the case of the CHO problem.

We also see that BIOCAT does not outperform literature on four sets of Pollen, Hela, Lymphoma and RNAi. It may be due to that reported results are based on current built-in algorithms in BIOCAT and Wndchrm’s longer list of descriptors including Halalick textures could have helped on these specific sets. Further effort on parameter fine tuning may also help improving the results. In this paper, we have been focusing on demonstrating the tool’s capability of empirical model comparison. Given the extensible design of BIOCAT, we expect to continue improving the accuracy as new algorithm modules suitable for the problems being added in.

On the other hand, it has also been pointed out that some low level descriptors could cause bias on classifying some biological images sets [34]. For example, CHO and Binucleate datasets can be biased in the sense that background artifacts could contribute in classification [34], so the accuracies may be optimistic. In general, it is worth noting that features and results should be validated in terms of biological relevance and best practice on this regard will be further studied.

In Additional file 2: Table S2, we demonstrate the 3D anisotropic wavelet feature for the K1503D image set, and how BIOCAT was used to do chain comparison. The anisotropic wavelet feature has rz set to 1 (3 neighboring z-slices are used). Wavelet level is set to 2. The nearest neighbor is a 3NN. Support Vector Machine classifier has a linear kernel and the regularization parameter set to 1. Fisher selector selects the best 500 features. As we see from Additional file 2: Table S2, the three algorithm chains containing 3D anisotropic wavelet (chains 1-3) deliver very satisfactory results. Both chain 1 and chain 2 achieve 100% accuracy with SVM as the classifier, regardless of whether Fisher’s feature selection is performed. Chain 3 has 90% accuracy with Random Forest as the classifier, while the alternative feature Hu Moments has a mediocre performance compared with Anisotropic wavelet even when the same type of classifier is used (Chain 4). This shows the potential effectiveness of the anisotropic wavelet feature as a 3D feature extractor. The bar chart that compares and reports the results of the five chains is directly from BIOCAT’s GUI, as part of its model selection functionality.

Figure 3 further exemplifies how BIOCAT selects an effective chain on some image sets in Additional file 1: Table S1. For each dataset, BIOCAT dynamically looks for a suitable combination of feature extractors, selectors and classifier, chooses the most effective algorithm chain for the given image set among candidate chains. For example, for the Binucleate set, the algorithm chain of “HAAR extractor; Fisher selector; 3-Nearest Neighbor classifier” out-wins other chains and gain 100% accuracy.

3D ROI classification and quantification

3D ROI quantification such as cell counting in high content images is a common problem in biology. Traditionally, it has been done using approaches such as connected component analysis or template matching without training a classifier. The general consensus on supervised learning using of a trained model has been that it can produce robust quantification results [15-19, 21, 35] and is suitable for large-scale analysis due to minimal user intervention [36]. Recent visualization tools such as Vaa3D has also made tagging of the 3D images more convenient (thus creating the training set is easier). BIOCAT can further facilitate pattern recognition based approach for 3D ROI classification and quantification.

Figure 4 shows an example of neurons in Drosophila m. (fruit fly) larvae nervous system, where gray clusters are the nuclei. Such images may contain thousands of cells per image. Often the cells form crowded clusters and the boundary among the objects may be blurry (see the zoomed area in Figure 4), which make such object counting in 3D biological images a challenging task.

We formulate the object counting as a pattern recognition problem based on voxel classification: For each voxel, we develop a model using BIOCAT to detect if it is potentially a center of a nuclei. A training set for ROIs on a training image with associated labels (e.g. positive and negative ROIs) needs to be generated. The process for 3D ROI classification then starts with loading the image and labeled ROIs into BIOCAT for running the model selection. Once the model is picked, new images are loaded into BIOCAT, ROIs are classified. Classification can be limited to images’ local maxima if needed.

Additional file 3: Table S3 shows the example algorithm chains compared during the model selection process of BIOCAT for the case study. We labeled 560 positive voxels and 430 negative voxels in the image. The 3D ROI volume is 7*7*5. The rz in 3D anisotropic wavelet is set to 2. SVM classifier uses linear kernel with regularization parameter set to 1. Five-fold cross validation is used to report the results. Chains are built and compared by BIOCAT to select effective models for the classification of 3D ROI around a fruitfly nuclei center. We experiment the cases when only 3D Anisotropic wavelet features are used; only 3D Hu Moments features are used; and the combination of 3D Anisotropic wavelet and Hu Moments features are used.

Additional file 3: Table S3 shows that 3D anisotropic wavelet feature alone can effectively yield 98.6% recognition rate. On the other hand, if only 3D Moments are used as the features extracted from the 3D ROI surrounding the voxel, recognition rate is only 96%. The combination of both features can further improve the recognition rate to 99.2%. It again shows that the 3D Anisotropic wavelet may be an effective 3D feature extractor for biological image classification. Additional file 3: Table S3 also shows that the use of random-forest classifier or support vector machine yields about the same result. All the comparisons among algorithm chains listed in Additional file 3: Table S3 are done by BIOCAT and the bar chart is the output of the tool.

The predicted center candidates by BIOCAT can be exported. They may be used for further quantification purposes. For example, the cell counting application described in [17] does neuron counting by performing mean shifting on the detected centers to move them to the closest center of mass. The nearby centers are merged before yielding the cell count. The advantage of such cell counting based on ROI classification is that when the cells are largely clustered and the boundary are blurry (which can be common in biological objects such as cells), the approach can give a count without the need of segmentation, especially when the shape of the object is oval.

2D ROI classification and quantification

To demonstrate BIOCAT’s 2D ROI classification, we describe a case study of axon detection for neuron morphology profiling in this experiment, in which we employ BIOCAT to detect axon pixels from the neuronal image to assist quantification of neuronal dendritic territory.

Intricate morphology is a striking feature of neurons and plays an important role in functional analysis and quantification of neuronal systems [37]. Among the neuronal morphometrics, the territory occupied by the dendritic tree is an important measure of the neuron morphology that not only indicates defective shapes of potential (dys)function of neurons, but also serves as an important parameterization factor for further quantification analysis in mutant screening. We consider GFP-labeled lobula plate tangential cells (LPTCs) in the brain of the adult Drosophila m. (fruit fly) (Figure 5). To extract the dendritic tree, we first automatically detect and remove non-dendritic subcellular components (soma and axon) from the image. Upon removal of the axon and soma, the territory of the dendritic tree can be easily estimated. In this paper, we focus on explaining the use of BIOCAT for axon detection: Similar as 3D ROI annotation, the flow starts with tagging the ROIs for training, then BIOCAT employs a trained model to detect axon candidate pixels from the neuronal image.

A training set consisting of 4 categories of image regions are randomly extracted from two neuron images - one wild type and one mutant type. The training set includes 10 axons, 50 branches, 7 soma and 9 background image regions. We make use of BIOCAT, which performs comparison of various combinations of extractors, selectors and classifiers as explained in previous sections. The winner algorithm chain consists of 2D HAAR discrete wavelet as features, Fisher’s criterion that selects top 40 features, and the Support Vector Machine as the classifier. Once the model was built, it was applied to other LPTC images to automatically annotate image regions and export the regions identified as axon candidates. Figure 6b visualizes how BIOCAT can be used to annotate axon, dendrites, soma and background. It is an overlay of the annotation results on the original image: The original neuron was light green; The axon and soma pixels are annotated using bright green and blue, respectively. A sliding window algorithm with interval parameter being 1 pixel is used in annotation. The axon candidates detected by BIOCAT will then be post-processed by finding the area using length and orientation considering that the axon is a long tubular structure. Once the axon (and soma) is removed, a rolling-ball algorithm is done to estimate the territory of the dendritic tree. Figure 6c shows the results of extracted dendritic territories from wild type and mutant neurons.

In this experiment, BIOCAT plays a similar role as segmentation, except that most segmentation tools deal with two classes (foreground and background), while the ROI classification gives multiple classes to annotate a pixel/voxel. We used a Fiji plug-in, “trainable segmentation” [38], to compare the results of the multi-class annotation for the given image. The plugin uses a trainable model (a fixed set of features and random forest classifier) to classify each pixel of the image. Figure 6d shows its output image, where the 4 gray levels, from deep to light, represent soma, background, dendrite and soma, respectively. We can see that although the plugin can also detect soma, it is less effective in separating axon from dendrite (many axon pixels are classified as dendrites). BIOCAT can extract the entire axon (the bright green region in Figure 6b. We can also see that BIOCAT classifies the background pixels accurately as indicated by a cleanly labeled background region (because Figure 6b is an overlay of output and original image, the annotated background shows as gray).

Adaptive algorithm chain selection

BIOCAT provides a flexible and adaptive platform that accommodates the ever-growing image samples and emerging pattern recognition and machine learning algorithms. Table 2, Additional file 2: Table S2 and Figure 3 show that different algorithms and their combinations indeed have different suitability for various problems. For example, in Figure 3, we see that while the random forest classifier does not perform ideally for the Binucleate set, it is effective for the Rat LiverGenderCR set (both with default parameter setting). Similar variations of efficacy can be found in feature sets.

The advantage of such an adaptive design is to find a good set of features for a given problem, without paying the cost of extracting a huge set of features or using a unsuitable classifier. Current related tools exploit two major approaches: 1) Use a fixed model including a feature set of medium size; 2) Start with a huge feature set, gaining accuracy at the cost of computational complexity. The former typically works better on some problems but not others. The latter can be an overkill on some cases and not suitable for large scale images due to computational complexity. For example, Wndchrm extracts thousands of features for any problem, among them Zernike features has a factorial complexity due to its radial polynomial coefficients. Some features are extracted using transforms based on the result of other transforms which are also very computationally demanding.

BIOCAT takes a different approach. It provides the platform of adaptively building different models for different problems, which may lead to fewer feature descriptors that are yet effective for classification. Note that such model selection is different from the computational selection of feature columns using statistical or other criteria as what Fisher’s (or similar) feature selector would do. It is because feature selectors just select indexed columns for a large matrix which may belong to different extractors. As a result, the annotation/testing stage would still need to start with all the feature extractors that are involved, which can negatively impact the performance for large images/sets. On the other hand, such features selectors can of course be part of an algorithm chain for BIOCAT, to further reduce the bandwidth for the classifier. Such approach also brings a better interpretability, which may potentially lead to a reversed way to understand some mechanism of differentiation among different biological structures or developmental stages.

BIOCAT also allows the selection of a suitable classifier as part of the algorithm chain, which few other tool for biological image classification offers (see Table 1). Instead of having computer scientists comparing (and often re-implementing) various classifiers, it provides biologists a GUI-based and objective approach to find a good pattern recognition model. BIOCAT GUI also allows to build the algorithm chains incrementally, to save/load chains and for generating reports after the chains are compared.

Speed and computational complexity

BIOCAT is able to process the datasets in Additional file 1: Table S1 within a reasonable period of time. The speed ranges from several seconds to several minutes with the sets on a typical PC (Intel Core i5 2.67 GHz 8 GRAM).

At the framework level, BIOCAT is designed to be memory-aware. Specifically, BIOCAT’s built-in feature extractors run on the image set in sequential fashion to reduce the memory footprint. In the case of 2D image sets, one image is loaded at a time. In the case of 3D images, one slice of an image is loaded at a time. After features are calculated, the same memory space is reused by the next image/slice. Since only the most current stack slice of the most current image is cached in memory, BIOCAT has a low requirement of computer memory at feature extraction stage. After feature extraction, subsequent feature selection and classification’s memory requirements are affected by image set size and feature set size (but not the original image size). We need to point out that individual algorithms have different levels of memory requirements and some third-party algorithms (e.g. Weka classifiers) may require bigger memory to process large sets. The speed of model comparison is linear with the number of models to be compared.

Multi-label annotation

In Experiments and Results Section, we demonstrated annotating an image by classifying ROIs surrounding a pixel/voxel. In addition, an entire image can also be automatically annotated, by formulating the task into a pattern recognition problem. Such annotation often involves multi-labels per image since an image may have multiple tags to be assigned. Annotation of biological images has traditionally been manual work, done by domain experts in a labor-intensive way. Recently, automated annotation has been attempted [16, 25, 26, 39].

The difference between multi-label annotation and the usual image classification is that typically an image classification task is single-objective (e.g. an image or a ROI corresponds to one and only one category in classification), whereas an automatic annotation task can be multi-objective. For example, fruit fly embryonic gene expression involves the annotation of multiple developed body parts in each gene expression image [16]. For such cases, BIOCAT chooses to formulate the annotation task as multiple binary problems, with each problem focusing on whether or not a label is present. Such labeling can be described using a text file (“target file” as mentioned in 3D Anisotropic Wavelet Texture Feature Section), which associates a training image with multiple labels. Model selection can be conducted for each sub-problem individually. During annotation, the collection of learned models, each for a label of interest, will be applied to the testing image. BIOCAT’s reporting tool outputs a consolidated annotation report, summarizing each testing image’s labels, ranging from zero to many. BIOCAT’s website shows an example and screenshot of such multi-label annotation task.

Connection with other tools and limitations

BIOCAT is designed with different purpose from tools such as ImageJ (for image processing) or CellProfiler (mostly for 2D cell segmentation/measurement). It does not reinvent the wheel. Instead, it focuses on the area where current tools are lacking, and meets the need of adaptable and extendable pattern recognition algorithms for image and ROI classification. On the other hand, they can certainly work together to complement each other. For example, ImageJ can be used to label ROIs in order to generate training sets for BIOCAT, or for preprocessing before the images are being classified. Other visualization tools such as Vaa3D can also be used for 3D ROI labeling and visualization, to complement BIOCAT.

ROI annotation is not the same as segmentation: For example, in the 3D ROI classification example, the cell counting may be done without knowing the exact boundaries of objects. Segmentation in a traditional sense is typically the issue of separating foreground from background and often unsupervised, while BIOCAT can work with multiple classes for the purpose of labeling/annotation. Segmented images by other tools such as ITK (Insight Segmentation and Registration Toolkit) can be used as input for classification by BIOCAT, while BIOCAT’s detection results may also be post-processed to deliver results similar as segmentation when every pixel/voxel is annotated.

BIOCAT is also different from general sense pattern recognition tools in that the built-in algorithms pay more attention to efficiency and memory since an image typically contains large number of pixels which is the raw dimension of data. It also has other functionalities specifically related to biological images. For example, if the image contains multiple channels, as is often the case with florescent stained microscopic images, BIOCAT can work on a selected channel. It can also limit the annotation of images to local maxima, useful for 3D ROI classification in fluorescent stained microscopic images. Its extensible design, on the other hand, can facilitate the inclusion of state-of-the-art algorithms in the fields of pattern recognition and machine learning, as they appear.

BIOCAT presents several limitations (or things it is not designed to do): There are no image processing algorithms in BIOCAT such as denoising or enhancement and the tool’s output often needs further post-processing to get results for the specific quantification. So BIOCAT is not intended as a replacement for either image processing or segmentation tools. Instead, it is for choosing a suitable supervised pattern recognition model, which can then be used as a discriminative model in classifying 2D/3D image sets or ROIs. The tool currently does not do exhaustive search of all combinations of algorithms for the consideration of computational feasibility. As the result, the selected chain is the best among all compared chains, but not necessarily optimal. It is also noted that some machine learning algorithms can be slow. When the user works with a large image set with high dimensionality, a GUI-based tool on a PC is not always the best choice. HPC version of BIOCAT for distributed model selection is currently under development.

Availability of supporting data

The software, along with supplementary materials can be found at http://faculty.cs.niu.edu/~zhou/tool/biocat/. The example images sets are available for download either from BIOCAT website (e.g. k150) or from the original provider’s site.