A high-level 3D visualization API for Java and ImageJ
© Schmid et al. 2010
Received: 13 January 2010
Accepted: 21 May 2010
Published: 21 May 2010
Skip to main content
© Schmid et al. 2010
Received: 13 January 2010
Accepted: 21 May 2010
Published: 21 May 2010
Current imaging methods such as Magnetic Resonance Imaging (MRI), Confocal microscopy, Electron Microscopy (EM) or Selective Plane Illumination Microscopy (SPIM) yield three-dimensional (3D) data sets in need of appropriate computational methods for their analysis. The reconstruction, segmentation and registration are best approached from the 3D representation of the data set.
Here we present a platform-independent framework based on Java and Java 3D for accelerated rendering of biological images. Our framework is seamlessly integrated into ImageJ, a free image processing package with a vast collection of community-developed biological image analysis tools. Our framework enriches the ImageJ software libraries with methods that greatly reduce the complexity of developing image analysis tools in an interactive 3D visualization environment. In particular, we provide high-level access to volume rendering, volume editing, surface extraction, and image annotation. The ability to rely on a library that removes the low-level details enables concentrating software development efforts on the algorithm implementation parts.
Our framework enables biomedical image software development to be built with 3D visualization capabilities with very little effort. We offer the source code and convenient binary packages along with extensive documentation at http://3dviewer.neurofly.de.
Life sciences are experiencing an increasing demand for scientific image processing. Images are the primary data of developmental and cell biology. The number of images is exploding with the availability of high-throughput and high-resolution technologies. The acquisition of large three-dimensional (3D) data sets, often as time series (4D), has become the new standard.
The first step in the analysis of biological image data is its visual inspection. In addition to the general requirement for visualization, the unique characteristics of each data set may demand specialized analysis. The development of novel analytical tools is greatly facilitated by the existence of well-documented software libraries. These libraries must provide (1) means to load and save any of the large diversity of image file formats; (2) implementations for computer vision algorithms; and (3) graphical user interfaces for data access by a human operator.
We have identified a lack of accessible 3D/4D visualization software libraries for biological image processing. Numerous image processing packages exist, either commercial (Amira, Visage Imaging; MeVisLab, Mevis; Imaris, BitPlane; Volocity, PerkinElmer) or open source (VOXX, ; VTK and VTK-based applications such as Slicer3D, BioImageXD, and V3D ; UCSF Chimera ; VolumeJ  and Volume Viewer ). These packages offer excellent solutions for the specific problems they were designed to solve. While end-users benefit from well-documented, special-purpose commercial applications, the development of custom analytical tools is better handled by open source packages. The application programming interfaces of existing packages range from the non-existent for most closed commercial solutions, to the very detailed and comprehensive open source VTK environment.
We have created a software library for 3D/4D visualization, with functions for surface extraction, volume rendering and interactive volume editing. Our library removes all the complexity of creating and interacting with image volumes and meshes in a 3D environment. We have designed our library to enrich the core functionality of ImageJ (and its descendant Fiji ), an open source image processing application. Via ImageJ, our library has access to hundreds of biological image file formats. Over the years, the scientific community has contributed a very large number of ImageJ extensions, known as plugins, which provide readily accessible implementations of numerous computer vision algorithms. With our library, we empower the ImageJ scientific community to rapidly implement custom analytical tools for 3D/4D data sets, with a minimal investment of time and resources in handling the complex details of a hardware-accelerated 3D environment. This reduction in the difficulty of visualizing 3D information commoditizes the usage of a 3D scene. For example, our library enables software developers to visually assess the correctness of individual algorithmic steps, such as the 3D shape of a mesh deformation. The simplicity of our library is in stark contrast to existing libraries such as VTK, which require detailed knowledge of the underlying data structures.
An example application in need of effective 3D visualization is the image volume reconstruction from electron tomography data as provided by TomoJ , an ImageJ plugin. Our library complements TomoJ, enabling convenient 3D analysis of the results without the need of external software such as Chimera . Many other plugins for image registration or object segmentation are similarly using our library for integrated and interactive 3D visualization.
For efficient rendering in computer graphics, we chose Java 3D: a low-level hardware-accelerated software library. Java 3D has the further advantage of being implemented for Java, thus enabling Java applications like ImageJ to interoperate with the graphics card of a computer, via either OpenGL or DirectX low-level native layer. Java 3D provides a fine-grained representation of a virtual scene as a directed acyclic graph . Operations on any node of the graph affect its entire subtree. In practice, this means that, for example, zooming in and out of a scene is expressed as a scaling transformation in a low-order node. High-order nodes encapsulate images and meshes.
Java 3D provides a collection of object templates, referred to as classes, each of which represents a node of the scene graph. The most relevant classes are:
BranchGroup: A node capable of being the root of several subtrees.
TransformGroup: A node that transforms the spatial representation of its enclosed subtree.
Switch: A node to toggle on and off the visibility of its subtree nodes.
Shape3D: A node representing a displayable object. The visualization of a Shape3D is specified by its Appearance and Geometry.
Geometry: Defines the geometry of the corresponding Shape3D, i.e. its vertex coordinates.
Appearance: Defines several attributes of a Shape3D regarding its visualization, such as color, transparency and texture.
The core ij3d module contains two principal classes, namely Content and Image3DUniverse. The Content class is a high-level representation of a single element in the 3D scene, such as a volume rendering or an isosurface. The Image3DUniverse (1) represents the 3D scene itself; (2) provides simplified access for adding, editing, selecting and removing Content instances to and from the scene, and (3) controls the view transform that represents zooming and panning. Via its superclass DefaultAnimatableUniverse, the Image3DUniverse also provides methods for 3D animation and recording movies.
In addition to data elements, the 3D scene can also contain analytical elements such as annotations in the form of named landmark points. These are added either interactively, or programmatically by accessing a Content instance.
As mentioned above, all elements of the 3D scene are related in a graph structure. Our constructed Java 3D graph links image objects (as Content instances) by wrapping them in ContentNode objects. The latter extend the functionality of basic Java 3D BranchGroup class, to serve as high-level scene elements. The ContentNode class is abstract; the four classes VoltexGroup, MeshGroup, OrthoGroup and CustomNode respectively represent volume renderings, surface renderings, orthoslices and custom geometries.
In summary, our library provides the means to instantiate a 3D scene with a simple user interface for interactions such as zooming, panning, editing objects and recording movies. Programmatically, the task of adding content to the scene has been reduced to a handful of lines of code (see listing 1), which is in stark contrast with the hundreds of lines of code required to achieve the same result using Java 3D directly.
We outline the features of our 3D visualization framework. We then describe its usage via both a graphical user interface (GUI) (for end-users) and an application programming interface (API) (for programmers).
The 3D scene is a virtual 3D space in which image volumes and meshes are displayed. Biological image volumes in the form of stacks of 2D images are shown within the 3D space in one of three ways: as a volume rendering, a mesh, or an orthoslice set. Volume rendering  is a technique for displaying image volumes directly. An arbitrarily-oriented image volume is projected to the screen with a transfer function such that dark pixels are more transparent than bright pixels. Meshes are constructed by applying the marching cubes algorithm  to image volumes to find a surface that encloses all pixels above a desired threshold value. Finally, orthoslices represent three perpendicular and adjustable planes that cut through the volume. An example of each type is shown in Figure 1. The 3D scene is capable of simultaneously hosting multiple image volumes, meshes and orthoslice sets. Each represented image volume has several adjustable attributes such as color, transparency and a local 3D transformation.
ImageJ's toolbar offers a collection of region of interest (ROI) tools. Closed ROIs, like rectangles, ellipses and polylines are used for interacting with image volumes (see "Volume editing" below). The point tool adds 3D landmarks, which are represented as small spheres.
The 3D scene can display landmark annotations for each image volume. These are added using the point tool. Existing landmarks are listed in a table that allows the manipulation of their properties, such as name and color. Each image volume hosted in the 3D scene may have an associated set of 3D landmarks of this type. A set of landmarks may be stored in a file for analysis, and reloaded in subsequent annotation sessions.
Two sets of homonymous landmarks positioned over two corresponding image volumes can be used for estimating a rigid transformation model  (see also Additional file 1, Figure S1). Using this model, one image volume can be aligned onto the other. The "Transform" menu offers options for exporting the transformed image volume as an image stack suitable for further processing with ImageJ.
The 3D viewer offers an option to record a 360-degree rotation of any 3D scene. Additionally, a recording mode is available. When this is activated, every manual rotation, translation and zooming of the display or any of its elements is recorded; when stopped, the recording is displayed as an ImageJ stack. Recordings may be output as videos via ImageJ.
Beyond the three basic image volume display types (volume rendering, mesh and or-thoslice set), the 3D scene accepts custom-crafted meshes. These meshes are typically generated programmatically, such as by automatic segmentation of image stacks.
Time-lapse recordings of 3D data sets can be loaded and visualized in the 3D scene. Standard command buttons for play, pause, fast-forward, etc. control the time point displayed in the viewer. Interactive zooming, rotation and panning are enabled as the time sequence progresses. When paused, the visualization of the current time point may be annotated, interacted with and measured as with any other 3D scene.
Our 3D visualization library includes a fully-functional plugin for ImageJ named "3D Viewer". The plugin is listed automatically in ImageJ's plugin menus. When executed, the plugin creates a new 3D scene, and automatically offers a dialog for displaying any open image stack as an image volume. The dialog provides the means to alter the attributes of the image volume, such as its representation type (volume rendering, isosurface (mesh) or orthoslices), and its color and transparency settings. The menu of the 3D scene window offers options for inserting further image volumes and editing, annotating and transforming them. Extensive documentation is available online http://3dviewer.neurofly.de, along with video tutorials and a 'Frequently Asked Questions' section.
Our framework exposes a public API to allow applications to integrate its features. A basic example demonstrates the use-case of visualizing in 3D an image volume and a mesh (see below). The example illustrates the development of an image segmentation algorithm, which extracts the boundary of the structures of interest as surfaces and represents them as a mesh composed of triangles. First, the image volume is rendered as orthoslices. Then the surface is displayed.
Image3DUniverse univ = new Image3DUniverse(640, 480);
ImagePlus imp = IJ.openImage("flybrain.tif");
Content c = univ.addOrthoslice(imp);
Alternatively, instead of addOrthoslice(), addVoltex() or addMesh() could be used to display the image as a volume or isosurface rendering, respectively.
List<Point3f> vertices = createVertices();
CustomMesh cm = new CustomTriangleMesh(vertices);
univ.addCustomMesh(cm, "triangle mesh");
The result looks similar to Figure 5b, which shows a confocal image of a fly brain together with parts of the surface of the medulla and the lobula (two compartments of the optic lobe).
Another API example illustrates a simulation of dendritic growth (Figure 4 and Additional file 1, section 2). The source code uses direct edits of the volumetric data to represent the growth over time. Documentation in the form of source code examples is available online at http://3dviewer.neurofly.de, in the Developer HowTos category. The documentation demonstrates in a tutorial style the available functionality of our framework.
Numerous ImageJ-based applications currently use our 3D visualization library. We briefly discuss below how three key applications use our library, illustrating the breadth of functionality we provide. We then conclude with future perspectives considering new demands for image processing and visualization in biomedical research.
The Simple Neurite Tracer  is an ImageJ plugin for semi-automated tracing of neurons in 3D image data. The application provides semiautomatic segmentation of filament-like structures such as neural arborizations and blood vessels. A starting point is chosen and then the filament is auto-traced up to a desired end point. The traced 3D path is visualized using components of our framework (Figure 5c). This example demonstrates how an analytical tool for measuring complex 3D structures can be augmented with 3D visualization capabilities to display those objects.
An algorithm has been developed for registering images of a 3D sample, where each image volume represents a different angle of view obtained by Single Plane Illumination Microscopy . The implementation of this complex algorithm required the 3D visualization of intermediate and final image registration steps. Our library enabled the algorithm developers to generate the required visualizations with very little effort (Figure 5d).
TrakEM2 is an ImageJ plugin for visualization, analysis, segmentation, reconstruction and registration of very large 3D image data sets obtained by serial section electron microscopy . TrakEM2 makes extensive usage of our framework for interaction with the 3D representation of image volumes and segmented objects of interest. The development of our library empowered TrakEM2 developers to plan and design for 3D interactive features that would not have been possible otherwise. Reciprocally, the high-performance requirements of TrakEM2 drove implementation of parallel processing strategies for isosurface extraction and mesh composition in the 3D scene.
The interaction of our library with other software packages, each with specific requirements, promotes the development of new features and improves performance. These improvements then propagate back and enhance other ImageJ applications. Detailed information about the use of the program and downloadable example code are available on our web page at http://3dviewer.neurofly.de.
The advent of high-throughput microscopy has increased the number and size of biological image data sets in need of analysis. The acquisition of 4D data, such as from laser-scanning fluorescent microscopy of cells moving through space, has become commonplace. Interactive data analysis of 4D data sets for object motion tracking is in increasing demand. Our framework contains all the key ingredients for 4D visualization and 4D data representation. For the near future, we will target the addition of convenient analytical tools that consider the time dimension. The ease of use and open source nature of our library enables the development of custom solutions for the highly specialized needs of biomedical image analysis.
In this paper, we introduced a new high-level 3D visualization framework for ImageJ. The framework provides an interactive 3D scene for image volume visualization, annotation, segmentation and transformation. For programmers, it offers the means to trivially augment the capabilities of their custom applications with hardware-accelerated 3D visualization. The framework has been very well received by the ImageJ user and developer community, and is currently in use by numerous ImageJ-based applications.
Project home page: http://3dviewer.neurofly.de
Operating systems(s): Platform independent
Programming language: Java and Java 3D
Other requirements: ImageJ
Any restrictions to use by non-academics: none
A JAR archive containing the software (with source code) can be downloaded from the project home page, following the Download link. Java 3D is available from https://java3d.dev.java.net and ImageJ from http://rsbweb.nih.gov/ij. The easiest way to set up these components is to install Fiji http://pacific.mpi-cbg.de, which bundles these dependencies and the software we present here. Additionally, a movie demonstrating its basic usage is provided as Additional file 3, and the software in its current state as Additional file 4.
We would like to thank Henrike Scholz for the confocal stack of the Drosophila thorax.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.