Features
We outline the features of our 3D visualization framework. We then describe its usage via both a graphical user interface (GUI) (for end-users) and an application programming interface (API) (for programmers).
The 3D scene
The 3D scene is a virtual 3D space in which image volumes and meshes are displayed. Biological image volumes in the form of stacks of 2D images are shown within the 3D space in one of three ways: as a volume rendering, a mesh, or an orthoslice set. Volume rendering [9] is a technique for displaying image volumes directly. An arbitrarily-oriented image volume is projected to the screen with a transfer function such that dark pixels are more transparent than bright pixels. Meshes are constructed by applying the marching cubes algorithm [10] to image volumes to find a surface that encloses all pixels above a desired threshold value. Finally, orthoslices represent three perpendicular and adjustable planes that cut through the volume. An example of each type is shown in Figure 1. The 3D scene is capable of simultaneously hosting multiple image volumes, meshes and orthoslice sets. Each represented image volume has several adjustable attributes such as color, transparency and a local 3D transformation.
The toolbar
ImageJ's toolbar offers a collection of region of interest (ROI) tools. Closed ROIs, like rectangles, ellipses and polylines are used for interacting with image volumes (see "Volume editing" below). The point tool adds 3D landmarks, which are represented as small spheres.
Volume editing
Programmatically, our library provides access to the values of all voxels in an image volume. Changes to voxel values are propagated to the screen. We use this feature for simulating the dendritic growth over time in the thorax of a fruit fly Drosophila (Figure 4). More material about this aspect is available in form of source code (Additional file 1, section 2) and a movie (Additional file 2).
Additionally, volume editing is possible interactively: The representation of an image stack in a 3D window enables 2D regions of interest to be projected across arbitrary axes of the volume. This enables what we refer to as "3D cropping", which consists of setting all voxels in the image volume that fall within the projected 2D region of interest to a particular color, typically black. We use 3D cropping to remove arbitrary parts of an image volume to inspect regions deep into the volume (Figure 5a).
Annotation in 3D space
The 3D scene can display landmark annotations for each image volume. These are added using the point tool. Existing landmarks are listed in a table that allows the manipulation of their properties, such as name and color. Each image volume hosted in the 3D scene may have an associated set of 3D landmarks of this type. A set of landmarks may be stored in a file for analysis, and reloaded in subsequent annotation sessions.
Landmark-based 3D rigid registration of image volumes
Two sets of homonymous landmarks positioned over two corresponding image volumes can be used for estimating a rigid transformation model [11] (see also Additional file 1, Figure S1). Using this model, one image volume can be aligned onto the other. The "Transform" menu offers options for exporting the transformed image volume as an image stack suitable for further processing with ImageJ.
Animation and recording
The 3D viewer offers an option to record a 360-degree rotation of any 3D scene. Additionally, a recording mode is available. When this is activated, every manual rotation, translation and zooming of the display or any of its elements is recorded; when stopped, the recording is displayed as an ImageJ stack. Recordings may be output as videos via ImageJ.
Custom content
Beyond the three basic image volume display types (volume rendering, mesh and or-thoslice set), the 3D scene accepts custom-crafted meshes. These meshes are typically generated programmatically, such as by automatic segmentation of image stacks.
4D Visualization
Time-lapse recordings of 3D data sets can be loaded and visualized in the 3D scene. Standard command buttons for play, pause, fast-forward, etc. control the time point displayed in the viewer. Interactive zooming, rotation and panning are enabled as the time sequence progresses. When paused, the visualization of the current time point may be annotated, interacted with and measured as with any other 3D scene.
Usage as a GUI application
Our 3D visualization library includes a fully-functional plugin for ImageJ named "3D Viewer". The plugin is listed automatically in ImageJ's plugin menus. When executed, the plugin creates a new 3D scene, and automatically offers a dialog for displaying any open image stack as an image volume. The dialog provides the means to alter the attributes of the image volume, such as its representation type (volume rendering, isosurface (mesh) or orthoslices), and its color and transparency settings. The menu of the 3D scene window offers options for inserting further image volumes and editing, annotating and transforming them. Extensive documentation is available online http://3dviewer.neurofly.de, along with video tutorials and a 'Frequently Asked Questions' section.
Usage as a programming library
Our framework exposes a public API to allow applications to integrate its features. A basic example demonstrates the use-case of visualizing in 3D an image volume and a mesh (see below). The example illustrates the development of an image segmentation algorithm, which extracts the boundary of the structures of interest as surfaces and represents them as a mesh composed of triangles. First, the image volume is rendered as orthoslices. Then the surface is displayed.
The first step is to instantiate an object of the class Image3DUniverse. Then we call its show() method, which creates a window to interact with the 3D scene. The scene graph is setup automatically.
Image3DUniverse univ = new Image3DUniverse(640, 480);
univ.show();
Next, the image volume is loaded. We display it as orthoslices in the 3D scene by calling the addOrthoslice() method:
ImagePlus imp = IJ.openImage("flybrain.tif");
Content c = univ.addOrthoslice(imp);
Alternatively, instead of addOrthoslice(), addVoltex() or addMesh() could be used to display the image as a volume or isosurface rendering, respectively.
If we assume that there exists an external method createVertices() that creates a list of points describing the vertices of the surface, and that three consecutive vertices define a triangle, the following source code shows how to create a custom triangle mesh and add it to the scene:
List<Point3f> vertices = createVertices();
CustomMesh cm = new CustomTriangleMesh(vertices);
univ.addCustomMesh(cm, "triangle mesh");
The result looks similar to Figure 5b, which shows a confocal image of a fly brain together with parts of the surface of the medulla and the lobula (two compartments of the optic lobe).
Another API example illustrates a simulation of dendritic growth (Figure 4 and Additional file 1, section 2). The source code uses direct edits of the volumetric data to represent the growth over time. Documentation in the form of source code examples is available online at http://3dviewer.neurofly.de, in the Developer HowTos category. The documentation demonstrates in a tutorial style the available functionality of our framework.