Image analysis pipeline HTPheno
There exist several tools supporting image editing, image processing and image analysis for many biological applications [6]. One popular tool is ImageJ [5], a flexible, open source image processing software based on Java. It comes with a graphical user interface and, with regard to scientific analysis, a collection of useful plugins and tools. Besides platform independence, the main reason for the wide distribution of the ImageJ software is the extensibility via Java plugins.
HTPheno is such a plugin and provides an adaptable image analysis pipeline for high-throughput phenotyping. With two built-in functions, (1) the calibration (HTPcalib) to specify different parameters for segmentation and (2) the automatic image processing, it can be used for analysing colour images in side view and top view. The HTPheno plugin realises automatic image processing for a number of images involving steps such as: region definition, object segmentation, display of the object extraction, morphological operation and compilation of the analysis results (see Figure 3). Finally, analysis results for all plants are comprised in a table and processing steps for each plant are visualised in an image stack.
Calibration of parameters for segmentation
Segmentation, which specifies if a pixel belongs to a defined object or not, is the essential and critical step in image processing. A colour image segmentation approach consists of the monochrome segmentation approach operating in different colour spaces. Some commonly used monochrome segmentation approaches are: histogram thresholding, feature space clustering, region based approaches, fuzzy approaches, neural networks or physics based approaches. They can operate in, for example, the following colour spaces: RGB, YIQ, YUV, I1I2I3, HSV, Nrgb, CIE L*u*v* or CIE L*a*b* [7]. Two fundamental questions arise in this context:
The colour image segmentation approach used here is multidimensional histogram threshholding: the histogram thresholding in combination of the two colour spaces RGB and HSV, which are appropriate colour spaces for images recorded by the high-throughput phenotyping platform.
Colour space
Based on the high correlation among the three primary colours red (R), green (G) and blue (B) colour image segmentation poses a challenge. If the brightness of the image changes due to changing or instable light conditions, all three colour components change accordingly. Hence we decided to use additionally the HSV colour space which is more intuitive to human perception. It separates colour information from brightness information. The human vision system can distinguish different basic colours (H, hue) easily, whereas the change of brightness (V, value) or purity of colour (S, saturation) does not imply the recognition of different colours [7].
By using nonlinear transformation the HSV colour space can be derived from RGB colour space. This means a linear change in H, S and V does not result in a linear change of RGB parameters. Therefore a slight change of input R, G, and B values can cause a large jump in the transformed H, S, and V values. Due to nonlinear transformation Hue has a non removable singularity and is numerically unstable at low saturation. If the intensity of the colour is close to white or black, Hue and Saturation play little role in distinguishing colours [7]. The HSV colour space alone is not sufficient for the segmentation of images recorded by the LemnaTec facility. Using a combination of RGB and HSV colour space results in a strong correlation of the calculated parameters obtained by HTPheno with the manually measured values (see section Validation).
Segmentation method
The irregular morphology of plants (here barley, Hordeum vulgare) restricts the development of a simple model which is at least necessary for the physics based segmentation approach. Images from the high-throughput phenotyping platform often have non-uniform illumination with the consequence of colour similarity between objects (such as carrier, conveyor belt, cages, sticks and shadows of them) and inhomogeneity within one object. Hence tests with an automatic segmentation method, the ImageJ plugin multi otsu threshold [8], an implementation of the otsu threshold algorithm to find up to 5 optimal threshold level (multilevel) of an image [9], does not deliver the required thresholds.
The best choice under these preconditions is to use a pixel based segmentation approach called multidimensional histogram thresholding (MHT). It utilises gray values of pixels without considering the neighbourhood. An image consists of areas in different gray level ranges. These areas can be separated in the histogram of the image by means of thresholds. Applied to colour images this approach operates in each colour channel histogram. An object is thus defined by a minimal and a maximal threshold for every channel of the RGB colour space and HSV colour space. To easily determine the thresholds for the object segmentation the function HTPcalib was developed. Images recorded by the high-throughput phenotyping platform contain beside the plant other objects. The user defines these objects by assigning them the correct colours in the image.
Some objects have a colour similar to the plant. Using this segmentation approach they would be segmented as plant as well. Therefore segmentation takes place in user-defined regions for top view images and side view images. A known object can only be situated within its defined region. Hence a set of regions can be defined. Here five regions are defined for top view images (region of soil, carrier, cages, sticks, and conveyor belt) and three regions are defined for side view images (region of carrier, cages, and sticks) since the cameras are installed in a fixed position (see Figure 3B, H). Once defined, regions grow and shrink automatically dependent on a user-defined scaling factor which depends on camera settings. Also a factor to translate pixel size into millimeter is set in HTPcalib.
After completing the calibration the automatic image processing for top view images and side view images can be applied.
Image processing
HTPheno retrieves single or series of images from the local file system and analyses automatically these images. Each processing step is visualised by an image (see Figure 3) first the original image is loaded (see Figure 3A, G), then the defined regions of the image are added (see Figure 3B, H) and after applying the object segmentation (MHT) a colour-coded image is shown (see Figure 3C, I).
Before performing the next step in the analysis pipeline the object of interest (the plant) is extracted (see Figure 3D, J). The plant has no defined region since plant parts can be located anywhere in the image. Some incorrectly segmented pixels and regions of the plant may occur because of colour similarity. To improve the segmentation of the plant the morphological operation opening is applied. Opening solves the problem by performing erosion followed by dilation. Opening removes small objects from the foreground (usually taken as dark pixels) of an image, placing them in the background and then smoothes objects.
The resulting refined segmented plant has a lower noise level (see Figure 3E, K).
Finally calculations based on the morphology result are performed. The visual analysis result is transferred to the original image and consists of plant outline; plant x-extent, plant y-extent and plant diameter (top view); plant width and plant height (side view). To get an impression of the plant size a scale bar (100 mm) is added to the image in the bottom right corner. All analysis steps can be checked by direct comparison of processed and non-processed images in an image stack per plant. A result table comprises all obtained phenotypic data: x-extent, y-extent and diameter in top view, width and height in side view as well as projected shoot area in both views for all plants. This result table can be exported into various spreadsheet applications for further processing.