Web tools for large-scale 3D biological images and atlases
© Husz et al.; licensee BioMed Central Ltd. 2012
Received: 29 September 2011
Accepted: 7 June 2012
Published: 7 June 2012
Skip to main content
© Husz et al.; licensee BioMed Central Ltd. 2012
Received: 29 September 2011
Accepted: 7 June 2012
Published: 7 June 2012
Large-scale volumetric biomedical image data of three or more dimensions are a significant challenge for distributed browsing and visualisation. Many images now exceed 10GB which for most users is too large to handle in terms of computer RAM and network bandwidth. This is aggravated when users need to access tens or hundreds of such images from an archive. Here we solve the problem for 2D section views through archive data delivering compressed tiled images enabling users to browse through very-large volume data in the context of a standard web-browser. The system provides an interactive visualisation for grey-level and colour 3D images including multiple image layers and spatial-data overlay.
Interactive browsing of arbitrary sections through large biomedical-image volumes is made possible by use of an extended internet protocol and efficient server-based image tiling. The tools open the possibility of enabling fast access to large image archives without the requirement of whole image download and client computers with very large memory configurations. The system was demonstrated using a range of medical and biomedical image data extending up to 135GB for a single image volume.
Multi-dimensional images are integral to biomedical research with many large scale imaging initiatives now in place to capture image data with genomic-scale coverage. These can be high resolution two- and three-dimensional (2D & 3D) and time varying images from the cellular through to whole organism level of resolution . This data acquisition has been matched by the development of image archive and database systems to support the curation and query of the data [2, 3]. Most of these systems will provide a number of visualisation options but in general these require the download of the full data-sets to be visualised using applications on the users’ workstation. In addition there are now a number of image databases that include a standard spatio-temporal reference framework onto which large volumes of data are spatially mapped to enable cross-comparison, query and analysis. Visualisation of these mapped data-sets could imply massive download of data beyond the local disk capacity and available bandwidth. In this context therefore the ability to access 3D image objects over a wide-area network such as the Internet is imperative.Recent work has demonstrated the clear advantages of tile-based image transmission and many zoom-viewers have been developed for example by Google Maps (http://maps.google.com), Zoomify (http://www.zoomify.com) and OpenZoom (http://www.openzoom.org). The key benefit is that the client application only requests enough data to display at the resolution required and therefore the data download scales in proportion to the view-window size which is in most cases constant. In addition the tile-based approach can benefit from data-compression, asynchronous download and the use of caching which makes very effective use of the network and provides a fast response for the user.
Some solutions involve the Internet Imaging Protocol (IIP), which is an open protocol that provides fast tiled delivery of large images through a multi-resolution image representation. IIP has been used in tele-pathology and educational archives . It allows a region of interest to be selected at any zoom level and provides efficient image transmission. An IIP server has been developed (part of IIPImage at SourceForge) as an open-source resource .
However, IIP (and all other related protocols) may only be used with 2D images and a similar presentation method for 3D objects did not exist. Therefore we have developed extensions to the IIP protocol, which we call IIP3D, and have implemented an open-source server to deliver this service based on the highly efficient Woolz image processing library [7, 8]. In addition we have developed a number of browser-based applications for specific projects which we use to illustrate the capabilities of the system.
The importance of virtual slicing systems for remote access of images was previously noted  and the IIP protocol was identified as a suitable interface for independent client-server applications. However the availability and flexibility of these imaging systems was limited by the proprietary and costly nature of existing implementations. Current image servers, such as BrainMaps.org, are able to deliver 3D data although they provide only predefined 2D sections. Glatz-Krieger et al. consider virtual slices only in the original focal planes of the biological material in the context of a 2D microscope slide. In this paper, we cut an arbitrary virtual section from the digitised 3D model. Such sectioning software exists, e.g. as commercial applications such as e.g. Amira  or open-source projects such as Slicer [11, 12], ImageJ  or our own applications MAPAINT and the Java atlas viewer, JAtlasViewer viewers . There are also online Java applications such as NeuroTerrain . Java is often selected for compatibility and platform independence, but sometimes falls short of this in practice.
Tile based image delivery, that transmits the target as smaller image blocks, is known from commercial web applications such as Google maps. This runs in any web browser and does not require additional software or an applet. We have developed a server based on the open IIP protocol that can deliver arbitrary sections through large scale 3D image data providing a very fast and efficient display that can be accessed with standard browsers. The Visible Human project has generated several Internet based image servers and clients. The EPFL server  is the most similar to ours. It is a high throughput parallelised sectioning server using a FastCGI (FCGI) web interface. However it does not allow tiled requests and has a proprietary protocol. In addition, for the delivery of section data, the EPFL server requires a high performance cluster. In contrast, the IIP3D server will run on standard Linux-based servers, is open source and has been tested under Linux, Mac OS X and Windows.
To supplement the viewing of arbitrary sections through 3D image data we have developed clients which can visualise any number of image layers including indexed graphical overlays. Many “atlas” views of data include the display of graphically defined regions that correspond to a segmentation of the image space. Typically these atlas regions are constrained in number (usually 255 in the context of an 8-bit index image) and constrained spatially not to allow overlaps. In contrast we have developed an index image that can have an arbitrary number of regions with any combination of overlapping regions and these are provided as an indexed overlay with complete control over the colour and opacity of the individual regions.
In summary, we have extended an open protocol and associated tools which allow fast presentation of 3D volumetric data as arbitrary cut sections delivered at the level of detail and localisation requested, in the form of compressed tiled images. This system is open source, generic and integrable with other functionalities. This extended protocol is IIP3D and in the next section we present the implementation, followed by details of the server and client software. We present results of performance testing with multiple simultaneous access and show how the client software can be utilised for a number of atlas-based and large scale volume image delivery.
Core to this work is the image processing library known as Woolz. Woolz is unusual in that it has separate data structures for the spatial domain of an image object and the image values, which can be 8, 16, 32 bit grey values, RGB α or float/double. The domain of an image object is simply the region (arbitrary point-set) of 2-D or 3-D space over which the image is defined. This form of record is particularly appropriate for biomedical images, as it allows a compact representation of images through not storing background values outside of the foreground object of interest and it encodes arbitrary regions of space (such as anatomical and gene expression domains) without the need for image values. Internally Woolz represents a 2D domain as a number of line segments or intervals for each line for which there is image information. In this way an arbitrary region of 2D space can be defined in a very compact form and binary operations can be reduced to a series of interval comparisons which are very efficient for binary-set, morphological and labelling operations. A 3D domain is simply a planewise stack of 2D domains and null entries for planes which are entirely outside of the domain.Each Woolz object has its own coordinate system defined with respect to a global origin of all objects and three coordinates k l and p for the columns, lines and planes respectively. The objects have uniform sampling and real world coordinates are achieved through a voxel size (spatial sampling rate for the voxels) which is stored in the domain. Because objects can be located at any coordinate location, including negative values, the domain and value table are defined relative to the overall image bounding box which allows for very simple re-location of image data. Note conversion from Woolz to more standard rectangular domain based image formats may lose this information. Converters exist to transform Woolz image objects to a wide range of other formats including NIfTI, VTK, Amira and standard tiff-stacks.
A recent extension of Woolz allows operations on large objects (e.g. up to terabyte volumes) that do not fit in system memory or would take many minutes to load from disk. It uses memory-mapped files, which reside on the external memory and are loaded on-the-fly block by block as the system requires it. 3D memory mapped objects are represented as a domain and grey value pair, however the grey values are stored in fixed size cuboidal blocks, accessed through a lookup table. The image value subdivision into cubes is similar to other solutions [16, 17] but uses memory mapping and file-system block alignment. These block sizes are integer multiples of the file system blocks enabling very efficient read and write operations. With solid-state disks the I/O rate is similar to data that resides on the main memory.
The viewer is fixed and the cut section is viewed by rotating about the line of intersection with the original z-plane. With non-zero pitch the effect of changing yaw will result in the section appearing to rotate.
A pre-defined or user-defined vector up is projected onto the cut section which is then rotated so that it will be parallel to the vertical axis on the user’s screen.
The fixed point used to define the centre of rotation can be coupled with a second fixed point to define a fixed line leaving one d.o.f. of rotation around that line.
Unconstrained viewing mode where all angles must be defined explicitly.
The IIP3D server is C++ software, based on a GNU GPL implementation of the IIP server by Pillay and Pitzalis . This is a FastCGI (FCGI) web server module that is managed by the web server (e.g. Apache) or as a standalone application and is connected by the web server as requests arrive.
The IIP3D server requests are parametrised HTTP requests and typically have four components: Server address: identifies the IIP3D server. Here we use the HTTP protocol and therefore the address is the URL of the FCGI application e.g. http://<serverIPaddress>/wlziip.fcgi. Resource Specifier: the second component of a request defines the image rewssource from which tiles are requested. This for pyramidal TIFFs is FIF , while for 3D Woolz objects is the WLZ command. View parameters: specifies the parameters of the sectioning plane (e.g. angles, distance) and the image return (e.g. format and compression) as well as setting for overlay colours and opacities. Information request: defines the specific information required from the object and sectioning plane set. This can be a tile, a full section or parameters of the object or sectioning plane (e.g. object dimensions, sectioning plane size, grey value or distance of a point from the sectioning plane, etc.).
The supported image request commands are similar to the IIP: returns the full image (jpeg or png); while JTL, PTL and TIL the jpeg-, png-compressed and uncompressed tiles. IIP3D also supports PNG format with lossless compression and alpha channel, both crucial for painted domain or textual overlays.
3D browsing is essentially the selection of a section by setting the various parameters defined by the PIT, ROL, YAW, DST and FXP commands. The IIP3D protocol also supports the viewing modes from Table 1 which given pitch and roll will define yaw.
IIP3D object request extension summary
Identify if IIP3D server is running
The size of the section
The size of a tile
The first and last plane, line and column number of the object
The 3D coordinates defined in 2D by the command
The range of the sectioning plane distance
The index value of a compound object that has foreground pixel at a point specified either the or the command
The grey or RGB value of a point specified either the or the command
The pitch, yaw and roll angles of of the sectioning plane
Returns the the display coordinates of the projected 3D point defined by PAB and its distance from the sectioning plane
The voxel size of the object
The volume of the object
In a number of atlases, the histological image may have overlay layers presenting for example anatomical domains or gene expression patterns. These are analogous to satellite images overlaid with street map or traffic information. IIP3D provides this functionality with the select (SEL) command and uses the compound Woolz object format to capture the set of domains. This is an array of independent Woolz objects stored in a single file and each domain can be accessed by selecting the corresponding index. Any number of such indexed domains are allowed and there are no spatial restrictions, the domains can overlap within the context of the viewed section and will be displayed as a set of overlapping colours.
The most general syntax of SEL is SEL = i,r,g,b,a where is the selected objects index, while r, b, g, a are red, blue, green and alpha channels specifiers.The RGB components paint with the chosen colour the selected domain, or for grey/colour objects filter the image selecting only the desired colour components. Simplified syntax allow 1, 2 and 4 parameters with i; (i,a) and (i,r,g,b), with 255 as the default for the omitted parameters.
Grey level or domain components are either independently selectable or stacked in an arbitrary order to generate grey level images, semi-transparent mask images, or a combination. A sequence of arbitrary number of commands specifies the composition order with the alpha channel information to determine layer visibility.
The OBJ=Wlz-foreground-objects requests the index or image value at query point allowing augmented display (e.g. name & description of a domain) in the viewer application. Uniquely the compound object based representation allows multiple overlaying domains to be displayed and queried which implies that the index query will in general return a list.
Faster image delivery is supported by three caching levels. As in the original IIP server, the IIP3D server caches the requested tiles. If multiple users access the same data, or it returns to a previously visualised region then the tiles are not recomputed but fetched directly from the cache. 3D images are often large, and the disk-read time is a significant overhead. Therefore woolz image objects once read are cached for further operations. However, for very large, memory mapped objects only the domain is cached, and not the values. Sections transforms which provide a fast look-up-table encoding of the affine transform are also cached.
Biomedical images may easily be many GB in size, for example the Visible Human image is about 17.5 GB and data sets for electron-microscopy data now reach many terabytes . Most current desktop workstations do not have sufficient memory to load such images and are incapable of being upgraded to be able to do so. Even when a workstation is capable of reading such a large object into memory the time taken to transfer it from disk is usually inconvenient; with a disk based file system capable of transferring data at 40MB/s it would take at least 7.4 minutes to read the Visible Human image into memory.
If the image is part of a resource then downloading significant data is very slow and copying multi-terabyte volumes infeasible. In most cases however, only a small subset of the image data is needed for visualisation since the typical user will be restricted by screen size. For this case we have extended the IIP3D server to use the memory-mapped option for very large woolz images enabling access to images of tens or hundreds of gigabytes.
The woolz imaging system can convert from a range of 2D and 3D formats and are therefore directly supported by the IIP3D server. Supported formats include: Amira lattice, Stanford density, gif, Analyze, IPLab, Jpeg, BioRad Confocal, PNM, raw image data, Tiff, Nifti and vtk. To get the benefit of memory-mapped option for very large volumes and the domain overlays with a global coordinate system, it is necessary to convert to woolz format data.
A JSON configuration file identifies various system parameters including the location of the image and the IIP3D server. It also defines the parameters of all the layers associated with an image, including name, location of data and display order. If a tree control is to be used (e.g. for selecting overlay data), the structure and content of the tree is defined in a configuration file.
The browser tries to be efficient in terms of transmission bandwidth and client memory by: requesting tiles only for the visible region of an image; requesting tiles only at the current resolution; for a scrolled/panned object, only new tiles are requested; layers which are not visible are not loaded.
In the case where a large image archive it to made accessible via HTTP requests it is often not desirable to have the web-server also act as the image server. In addition with a large archive or high access rates the requests may need to be delegated to a number of servers. To provide a single point of access and enable this delegation to image servers separated from the Internet by a firewall, we have developed IIP3DProxy, an FCGI application, which filters requests and forwards them to a set of IIP3D servers. The communication conforms to the FCGI protocol. Though it was designated to work for IIP and Woolz requests, it is generic and can route any FCGI request, hence it allows chaining of multiple proxies.
The IIP3D protocol includes extension of the query commands to provide information for image layout and user feedback. These include the size (in pixels) of a given section image, the size of a tile, the voxel size, the object volume and bounding box and the grey or colour value at a given pixel location within the section image. For example, the voxel grey value query with respect to the 3D location (190,200,190) is http://localhost/fcgi-bin/wlziipsrv.fcgi?∖WLZ=/objects/TS18.wlz∖PAB=190,200,190∖OBJ=Wlz-grey-value which results in the reply Wlz-grey-value:73
This query can also be made with respect to the 2D location within a given tile.
Test data composition
A series of experiments were performed with a single IIP3D server and 1 to 60 clients. The server and client computers were all equipped with dual 3GHz Intel Xeon 5160 CPUs, 7200 R.P.M. SATA disks and 32GB RAM and are directly connected via a dedicated 1 Gb Ethernet switch. There was no other activity on the server, on the clients or on the network other then operating system management tasks.
For testing, we used standalone C++ executables started simultaneously on the 11 client nodes. For test cases with more clients than available client-nodes, multiple concurrent instances were started on the same client node.
The timestamp of each request was recorded along with all other IIP3D request parameters. This allows replication of user browsing behaviour. The object queried was already prefetched from the disk into the server main memory and the complete set of requests were repeated as whole series and timings captured. The first test set takes 4.88s, while the consecutive requires only 0.77s, suggesting the effect of the local tile cache.
Summary timing results (in seconds) for different access scenarios described in the text and for selected client numbers
Average response with cache
0.46 ± 0.06
Average response no cache
4.23 ± 0.11
It is interesting to remark in Figure 7b that the running time of the timed test sets recorded over 273 seconds, it is respected for the cached scenario, while for the non-cached scenario up to 20 clients managed to fit in the time user needed to browse through the object. This suggests that up to 20 clients may be served without major performance degradation, however as is shown above the display delay increases.
Client executables were started remotely from a test script. We had no control on the exact timing of these processes neither how the request are prioritised by the apache web server, therefore occasionally “lucky” clients occur with lower execution time then the average. Note the client executable did not use any local caching therefore the benefit of the browser cache is not included with these measurements which are therefore a lower bound on the performance experienced by the user.
IIP3D is configurable for the application in use. We present the basic 3D browsing functionality, followed by the support of handling overlaid 3D image data and large databases. We exemplify these functionality on the male VisibleHuman dataset , adult mouse muti-modality Waxholm data , HUDSEN developing human brain atlas , fly brain atlas  and the eMouseAtlas atlas models .
Figure 8b shows concurrent viewing of image layers with control of a colour filter and transparency across each image layer. The SEL command enables fine-grained control of colour and transparency of annotation overlays which can be considered as equivalent to an indexed image. An index defines a region of the 3D image space and the intersection of that 3D region with the current section will return tiles with those pixels given the requested colour and transparency. If two regions overlap in the same region then the colours will be merged. The result is a single layer that can be used to show regional annotations for example anatomical delineations or the presence of gene-expression. Pointing on the grey value brings up the name of the underlying single or multiple domain, if available. This is possible because the IIP3D compound object formalism allows annotated regions to overlap and there is no constraint in number.
We have tested this system in the context of atlas-based biological data however it is clearly applicable to a wide range of image data for which arbitrary re-sectioning is a useful visualisation. This visualisation is typically the core requirement in basic science but also medicine where views through the data are the primary views needed by experts who want to see “the real data”. It is complementary to recent development of in HTML5 in particular WebGL which is exemplified by the Google “Body Browser” demonstrator (http://bodybrowser.googlelabs.com). We plan to use IIP3D in a future WebGL based application to provide cut section image textures.
For archives of volumetric data or single very-large image volumes this server development is the only plausible mechanism to deliver views to users who do not have the required bandwidth, local storage capability or the compute capability to be able to download and view the data. Here we realise the benefit of low-bandwidth tile-based access to image data coupled with very efficient image-processing to calculate the virtual section tiles. This technology is open-source and will run on any unix-based web-server.
Through the IIP3D API the 3D viewer can be integrated into complex applications such as Next Generation Embryology2 - a 3D spatio-temporal framework that augments IIP3D data in conjunction with a repository to deliver research and educational material. Users may add supplementary augmentation (points or region) to the 3D volume, together with texts, videos, external links.
The compound representation of multiple images (i.e. grey level, colour or segmented domains) and providing them as individual or composed images allows compact representation of related data, and offers a novel 3D visualisation of overlays. Also, it has power of representing overlapping domains that standard index-image representations can not provide. We have shown that the performance of our implementation of IIP3D scales well with the number of parallel requests. Grey level or colour data is successfully augmented with domain or gene expression data. Controls, such as sliders, layer selection tool or locator resemble functionality of know tools from desktop applications. These tools and the associated tiling code are all available as open-source.
Future developments of this software include grey-level transforms such as range slicing for 16bit medical data, extension to vector and scalar data and the introduction of a standard glyph library for location and directional marking. In the context of the client interface we will address usability by undertaking a user-evaluation study.
Project name: Woolz IIP
Operating system(s): sever: Linux; client: platform independent
License: GNU GPL
Any restrictions to use by non-academics: according to GNU GPL
Internet Imaging Protocol
3D extended Internet Imaging Protocol
Fast Common Gateway Interface.
The authors would like to thank Tom Perry who developed the original version of the section viewer  and Newcastle collaborators Susan Lindsay and Janet Kerwin for letting us show the HUDSEN embryo data. We acknowledge support from the Medical Research Council, UK and NIH support under grant #1R01MH070370-01A2.
The Visible Male data is courtesy of National Library of Medicine, Visible Human Project, the CS17 embryo data is courtesy of the HUDSEN Electronic Atlas of the Developing Human Brain project, Newcastle University and the Waxholm data curtesy of the INCF (http://www.incf.org)
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited