Pattern recognition algorithms suitable for multiple-dimensional biological image classification and annotation are central components of BIOCAT, which include multiple feature extracting and selection algorithms, as well as classifiers. In this section, we will start with a brief summary of the adaptive design of BIOCAT that allows algorithm modules to work together and form a customizable solution for a given problem. We will then describe the representative algorithm modules (extractors, selectors and classifiers). We will also detail a 3D feature extractor.
Design of BIOCAT
BIOCAT is designed with 1) usability, 2) extensibility, 3) inter-operability, and 4) portability. Figure 1 shows some screenshots of the BIOCAT GUI.
Usability
BIOCAT is intended to be mainly used by biologists with no much experience in pattern recognition (although technical users might also find it useful for algorithm evaluation). While some training is required, we expect that the end users can adaptively build a model that is best fit for their classification problem at hand, facilitated by the GUI of the tool.
As shown in Figure 1, BIOCAT has a user-friendly GUI that permits users to conduct: a) customizable model building, which adaptively selects a model for the given biological image classification/annotation task. The user can do so using a training set and a testing set, or cross validation on a given set. The model can be built, compared and saved. One important feature of BIOCAT is its “multi-chain comparison mode” (see Modularized algorithms and algorithm chain Section), in which multiple algorithms chains are compared for the given problem. b) Annotation and classification of image sets or ROIs, using the chosen model. BIOCAT can work with both image classification or multi-label annotation: If each image in a set corresponds to one label, then it resembles normal image classification; If each image could correspond to more than one labels, then it is a multi-label annotation task.
In order to facilitate supervised learning, the training images (or regions) are associated with corresponding labels. BIOCAT allows three possible ways of such association and the user can pick the most suitable way for him/her: a) target file input mode: a text file that assigns a label or multiple labels to each image; b) directory tree input mode: a directory structure with each subdirectory being an image category; or c) ROI mode: ROI/landmark files where the name of the ImageJ ROI zip file or Vaa3D landmark file indicates the image category.
Extensibility
BIOCAT has a modular design, and thus is extensible and developer-friendly. BIOCAT algorithm specification is provided for extending BIOCAT with more algorithms: An algorithm developer can add a new algorithm to BIOCAT as long as it conforms to the API (Application Programming Interface) specifications for a feature extractor, a feature selector, or a classifier. An XML file can be used by the module developer to specify parameters of the algorithm that will show up on GUI. (See Developer’s Guide on BIOCAT website for tutorial).
Inter-operability
BIOCAT runs as a standalone application that focuses on classifying/annotating a set of images (or ROIs in them) using pattern recognition algorithms. It can inter-operate with other tools with different focus. The ROI manager in ImageJ and landmark manager in Vaa3D can be used to save ROIs and import them into BIOCAT for training. Classification/annotation output files can be edited to be Vaa3D landmark files for further visualization. BIOCAT also generates reports in PDF files for algorithm comparison results and annotation results.
Portability
BIOCAT is developed in Java. Thus this software is portable to various operating systems such as Windows, Mac and Linux as long as Java Virtual Machine is available.
To summarize, the uniqueness of the BIOCAT tool, other than the list of built-in pattern recognition algorithms suitable for image classification tasks (to be explained in next section), is that the algorithms are modularized so that they can be “chained” in a customizable way to form adaptive solution for various problems, and the plugin-based extensibility gives the tool an open and flexible architecture to include future algorithms.
Modularized algorithms and algorithm chain
The BIOCAT’s algorithms are modularized to facilitate the adaptive comparison of what we call “algorithm chains”, which is a sequence of several pattern recognition algorithms that include some feature extractors, one or more optional feature selector(s), and a classifier.
For feature extraction, we consider 2D and 3D textural, morphological and structural features. Texture features are an important category of effective image features such as wavelet features. Morphological features describe the shape and structure of 2D and 3D objects. A pixel in a 2D digital image consists of 4 or 8 neighbor pixels depending on whether the diagonal neighbors are considered. Some 3D features can be extended from 2D by extending an 8-connected neighborhood to a 3D 26-connected neighborhood. Examples of morphological features include Hu moments, which are statistical summaries of intensity in the 2D or 3D object neighborhood, and measures in relation to the center of mass of the images [28] or Zernike moments which are orthogonal invariant moments using a set of complex polynomials [29]. Other algorithms include Hessian features which are suitable for detecting tube-like neuronal structures, Gaussian derivatives and Laplacian features [30], as well as object statistics [21]. Currently 14 different feature extractors are available in BIOCAT with the following examples, each providing one to multiple measures extracted from an image:
-
8 2D Hu Moments, invariant to translation, scale and rotation.
-
8 3D Hu Moments: 3D extension of 2D Hu Moments;
-
Zernike moments: Default to the first 20 Zernike moments.
-
7 statistics for the objects in an image: number of objects; their average size and variance, and their spatial location in relation to center of mass of the image, both average and variance; ratio of the size of the largest object to the smallest; ratio of the distances to center of mass between the furthest and the closest.
-
Object territory: overall territory the foreground object.
-
Image’s Hessian and Laplacian features.
-
2D and 3D wavelet features. The 3D anisotropic wavelet texture feature will be detailed in 3D Anisotropic Wavelet Texture Feature Section.
BIOCAT has feature selectors such as Fisher’s criterion that select a subset of features. BIOCAT also provides several classifiers including support vector classifiers [31], nearest neighbor, Naïve Bayes, decision tree, and random forest.
As explained in Implementation Section, BIOCAT supports an extensible design, so the above algorithms are of a growing list. Currently, BIOCAT has about 20 algorithms for 2D/3D feature extractors, selectors and classifiers. The list includes the 3D anisotropic wavelet feature extractor and re-implementations of known algorithms (a majority of the feature extractors are re-implementations). The (re)implementations are memory- and efficiency-aware for working with high content multidimensional bioimages. These feature extractors in BIOCAT reuse the memory space for each image. In addition, due to the extensible design, libraries may be wrapped and incorporated as plugins whenever such algorithms become available. For example, Hessian features is a plugin wrapped around FeatureJ implementation [30].
Note that the “algorithm chain” in our paper refers to a sequential flow of feature extraction, selection and classification, which is different from the narrower sense of supervised (or unsupervised) learning of pattern recognition. When multiple feature extractors are used, these features are linearly aggregated. The combined feature’s dimensionality is the sum of all individual feature vectors. Such combination may lose some property (e.g. invariance) of certain features if they are not combined with features with similar property, yet it provides the flexibility for extending to many feature extractors that are often useful for biological image classification. When multiple selectors are used, it is a cascading design: the feature vector goes through the first selector. The resulting feature vector is then filtered by the second selector and so on.
With all the algorithm modules as building blocks, BIOCAT can conveniently build algorithm chains to compare multiple alternative models. One example algorithm chain can be an anisotropic 3D wavelet feature extractor along with a 3D Hu Moments feature extractor, followed by a Fisher feature selection and ultimately a support vector classifier. The chains with the same types of algorithms but different parameters are considered to be different chains, so that dissimilar parameters (e.g. multiple wavelets) can be compared to find the suitable ones for the model. Figure 1b shows BIOCAT’s screenshot of building and comparing the algorithm chains.
In addition, functionality is included in BIOCAT for ROI annotation (Here the term “annotation” refers to the computational task of automatically labeling a new image which is formulated as a supervised pattern recognition task):
-
1.
Sliding window-based ROI annotation. With a model learned and selected, BIOCAT can conduct automatic ROI annotation on a new image using a window sliding algorithm. A sliding interval parameter is defined which represents how many pixels apart will the trained model be applied to a pixel/voxel. Because it represents the frequency of decision making, a small interval leads to a higher annotation resolution with longer running time. When the interval is set to 1, then every pixel in the image is annotated: a ROI is extracted surrounding each pixel/voxel, and a classification is performed to assign a category to the pixel/voxel. The parameter can be adjusted via GUI.
-
2.
In the case of 3D image annotation, it is possible that a large portion of the 3D image is black. To improve annotation efficiency, local maxima can be calculated using morphological dilation. The decision can thus be made only on the set of local maximum voxels. Such scenarios are useful for 3D object quantification (objects are labeled based on the pixel classification of BIOCAT), such as synapse or cell counting using 3D confocal images, where objects of interest are the bright regions in largely dark images.
3D anisotropic wavelet texture feature
One important category of effective image features is texture features, which is a family of features that measure the texture of images such as wavelet features. In particular, wavelet features are obtained using discrete wavelet transform (DTW) with wavelet functions [32]. Two dimensional wavelet features had performed well in our previous work with gene-expression annotation of fruit fly [16]. However, full extension of these features from 2D to 3D can lead to a big increase in the number of features, and consequently the storage and computational need. For example, for wavelet transform, a full extension to 3D wavelet features will lead to a number of features that is cubic to the side length of the object (instead of being quadratic in 2D). We thus designed a multi-scale anisotropic 3D wavelet feature. Such features can be particularly suitable for 3D confocal microscopic images since it adapts to the anisotropic nature of confocal imaging where z resolution is typically less than x-y resolution.
The feature for a voxel of interest at (x,y,z) is extracted from its surrounding 3D volume:
where I is the intensity, the size of the extracted volume is (2*rx+1)*(2*ry+1)*(2*rz+1), ψ is the discrete Haar wavelet basis function, k, n are the dilation and translation factors of wavelet with k=0,1 (for two-level wavelets), n=0....(2rx+1)*(2ry+1), and (2*rz+1) is the number of slices included in calculation. w(.) is a Gaussian function with middle z slices weighted heavier than other slices.
The multi-scale component of the 3D anisotropic wavelet is represented by the dilation factor k of the wavelet transform. The number of the scales is also called wavelet level. At each level, a Haar-based discrete wavelet transform is done on the approximation image from the previous finer resolution. Figure 2 shows an example of level-2 3D isotropic wavelet, where 2*rz+1 slices are involved in calculating the feature for a 3D image of a fluorescently labeled fruitfly brain.
Note that other tools also use wavelet for image analysis. For example, Icy [11]’s spot detector plugin [33] uses wavelet adaptive threshold for object detection, which is an unsupervised scheme different from the method presented in BIOCAT.