NeuroTerrain – a client-server system for browsing 3D biomedical image data sets

Server manager

Administrative clients communicate with this module via a standardized XML stream over a TCP socket. Secuity can be handled using SSL, and/or control limited to a UNIX-domain socket, which restricts access to processes running on the local host.

Servers

This NtAS implementation actually consists of two linked servers, each with its own communication socket. One server implements a binary interface, used to retrieve sliced images of the dataset; in the other, XML data is used to return single-instance data items.

The binary server is designed to maintain a persistent TCP session connection, similar to the way telnet is implemented. Once the client logs in, a data structure maintaining atlas data set state is created and linked to the user via the same TCP socket (IP number/port number combination) used to connect to the server. Once the DM identifies the client and approves a connection, a list of atlases/datasets available for browsing is returned. The client requests the desired atlas, and the server returns the dimensions and resolution of the chosen dataset. Subsequent session communication between the client and the server include serialized requests for slice (image) data based on the client's specified plane orientation, and requests to show/hide any available segmentation, including presentation metadata.

The XML server uses a single-transaction oriented protocol much like http. The user logs in, is approved, and can then request information about the selected atlas, including 2D paths representing the intersection of a segmented volume and a specified slicing plane, etc. Per-client state data is tracked via a unique session ID which expires after a period of inactivity. This interface is an implementation of an inter-atlas interoperability standard developed within the Mouse BIRN subproject of BIRN[5] which was designed to promote atlas data sharing amoungst the Mouse BIRN neuroanatomical data management clients NeuroTerrain, SMART Atlas and SHIVA. Figure 1 provides an architecture diagram to support the description given below.

Slicers

The data slicer runs in an independent thread, and accepts slicing requests from a queue fed by the servers. There is no theoretical limit to the number of instantiated slicing threads, but there is generally no practical advantage to having more slicing threads than processor cores on the host computer. Each slicer thread is able to handle both image pixel requests from the binary server and segmentation path requests from the XML server. Results of the slicing operation are returned to the client by the slicer via a return queue.

The key to the real-time responsiveness of the Slicer component is the use of the Macrovoxel 3D data format which provides for an extremely fast nearest-neighbor-based assembly of any arbitrary place on or off the original imaging axis[1]. Both the pixel data sets as well as the associated segmented Volumes of Interest (VOI) are stored in an efficient format.

Binary slicing requests are serialized. This enables client to provide live navigation by tracking the mouse pointer, sending a continuing stream of slice requests as the mouse moves. A slicer thread checks each incoming slice request serial number against the current value in the client's state data; stale requests are dropped. Returned slices are also serialized; several slicing threads may be running, and two different threads may be processing requests from the same client. The client ignores stale slices returned. Region of Interest (ROI) path requests are not serialized, since the order in which they are returned is not critical to the way they are processed by the client and presented to the user. This enables the server to process ROI path requests more efficiently.

Data manager

The DM is responsible for loading, unloading and maintaining access to datasets, and associating ontologies with the datasets. It also manages user access permissions to these datasets.

Process queues

The process queues are used to communicate between the server threads and the slicer threads. Requests for slicing service are placed in a queue as a data structure including client ID and state information, target information, and requested action. Each slicer calls a PQM function that blocks waiting on a condition variable for queue state. Once a request is received, the slicer examines the request type, and dispatches it to the appropriate handler – binary or XML. This handler satisfies the request, and then returns the resulting data to the PQM return queue, where it is then forwarded to the client.

This module, implementing queues which the slicing thread feeds off, readily allows for multiple slicing threads, and also allows for threads to be started or stopped on the fly, if so desired, greatly enhancing the scalability of the server.

Server Interfaces

The NtAS interactions are encapsulated in the NT-SDK-NtAS-Comm framework. Within this library, the client communicates with the server via two discrete sub-interfaces. One, derived from the original MacOStat server, transfer all data in binary format – values are encoded as either 32 bit floats and integers or strings (atlas or region names, for example). Integer codes transmit NtAS request commands chosen to give recognizable display in a binary data stream. This interface provides efficient, dynamic image-oriented transactions: return of a sliced volume being the main example, but control of the VK is also included here. The binary commands of this socket protocol are implemented as a collection of Java classes handling all the communication details, making server easily available to external programmers by merely creating a new object of a particular command class and sending it to the communication manager's write command. A communication controller embedded in the NT to mediate server transactions,

The NT-SDK-NtAS-Comm also provides callbacks as a Java Interface to monitor NtAS responses to the binary commands – e.g., the latest slice image or associated metadata. By implementing these methods, an applications can dyanically react to new server responces as required.

The other request interface in the NT-SDK-NtAS-Comm library is based on XML-formatted text streams, and is used to poll atlas state: information such as the 2D planar location within a mounted data set, paths describing the intersection of VOIs with the VK, and so on.

Detailed specifications for these protocols can be found at the web site [24].

Graphical User Interface

The GUI consists of:

• the VK JPanel used to set the current 3D atlas cross-section view (Figure 2a);

This interactive functionality is embedded in the NT-SDK-NetOStat-GUI framework. By obtaining these JPanels from this library, an enclosing application incoporates the entire binary server API, as well as the GUI controls listed above. Detailed specifications for the NT-SDK API can be found at[24].

The knife is presented as an intersecting plane within a wireframe rectangular volume whose dimensional extent is the x, y, z bounds of the 3D atlas dataset under view (Figure 2a). A series of tool buttons enables an user to specify how mouse dragging over the wireframe afects VK movement. The investigater may translate the knife along the slicing axis, or rotate it about any of the three cartesian axes. A series of buttons in the SV control panel enables the user to select the cutting axis to present slices along any one of the standard anatomical axes – coronal, horizontal, or sagittal (Figure 2b). Thus, a user can slice through the 3D data set in an arbitrary plane either on or off axis. Each user intiated VK movement triggers a request to the server resulting in an updated atlas image in the SV (Figure 2c). A client preference setting enables an investigator to adjust the scaling of mouse cursor movement to knife movement on the server (default = 5 server slice pixels/1 client cursor pixel).

The NT-SDK-NetOStat-GUI also can provide the VK and SV interfaces as separate, floating frames and enable the researcher (or external SDK programmer) to create multiple NtAS connections. Each of these distinct SVs (Figure 3b) comes with a collection of tool buttons supporting easy view specific actions: switch slice axes, select VOIs, pan & zoom, save knife locations, and save atlas grayscale views. A single VK provides coupled navigation across all views (Figure 3a). This enables the researcher to simultaneously co-navigate multiple, aligned Macrovoxel data sets. From the point-of-view of external programmers, each SV is provided as a separate object not only providing access to the JFrame itself, but also the callbacks tied to each one (e.g., automatic updates to atlas slice axis, grayscale image, zoom level, etc.). All navigational interoperability amoungst the distinct SV connections and the VK JFrame is handled internal to the NT-SDK, though this activity can also be driven from the external program. The programmer can also request these views and the VK interface be passed as JPanels for tighter integration into the external program.

The multi-threaded nature of the server makes it possible to support these parallels data connections. Multiple NtAS instances can also be setup when user demand requires more slicing throughput.

A separate dialog allows the user to examine the list of volumes of interest or segmentated regions included in the atlas, select a subset for display and to specify a particular color for the volume (Figure 3a). These colored ROIs are superimposed on the main display either as a semi-opaque mask or as an outline (Figure 3b). A researcher can toggle individual ROIs on/off, or selectively dim certain ones to emphasize a selected region among a set of related regions.

A collection of hierarchical menus under the main NeuroTerrain Atlas Server menu supports the following, some of which are described in more detail in the following section on client-server interactions:

These features are all described in more detail below.

Login

Prior to performing any other action, the client must log in to the server. This allows the server to set up the necessary session structures, and registers the particular client implementation or user for any special actions desired. For example, at this point the user-agent name is transmitted by the client. On the server side, this can be used to select a particular default dataset, to specify that only a subset of metadata is returned to the client, or restrict the client to a subset of mounted data sets. The user agent is a configurable value in a Java properties file within the NT-SDK, so clients can be distributed to a user base with unique and defined set of default conditions the server can enforce. Using either the binary or XML API, the login message contains the user-agent ID, and optionally a user ID. Once the login message has been processed by either server protocol, an acknowledgment is returned. In the case of the binary API, the return message contains a list of all available 3D datasets (atlases or other Macrovoxel files), associated with a reference number. In the case of the XML interface, the returned message contains the unique session ID. A client logged in via the XML API can then send a query for a list of available datasets.

Atlas selection

Once successfully logged in via the binary API, the client further configures the session by choosing which available dataset to view. This results in the server returning a data block containing the dimensions and resolution of the requested dataset. When using the XML API, a client does not declare a specific atlas data set up front, as the desired atlas must be included in all subsequent requests.

Axis selection

After selecting an atlas, a client using the binary API must specify the axis along which the VK will be moved. This value defaults to the Z-axis, which provides the anatomically standard coronal tissue presentation The X-axis and Y-axis similarly provide sections in the sagittal and horizontal orientations, respectively. A coronal view of the NeuroTerrain mouse brain histological atlas is presented in Figure 2c.

Knife position change

Once atlas and axis selections have been completed, the system is ready to return sliced volume data. To initiate slice retrieval, the client sends a knife position request. This involves specifying a sequence number for serialization purposes, an offset along the slicing axis, and a triplet of angles describing the orientation of the VK to the slicing axis (angular rotation about the standard cartesian x, y, and z axes). The server queues this request for the slicer, along with client address information and the client sequence number. In turn, the slicer executes the slice requests, and returns pixel data as a series of linear pixel arrays or rasters, along with raster positioning information and the slicer's sequence number.

ROI request

A client using the XML API is able to request paths describing the intersection of VOIs with the VK. To do this, the client provides identifiers for the VOIs to be sliced, and the orientation of the VK. Presently, brain region names and abbreviations developed in our lab are hard-coded into a list and linked to the assocated VOI data set. We are in the process of incorporating a more dynamic means of associating anatomical segmented region identifers from a public knowledge source, such as NeuroNames[25] in the case of brain region volumes. A complete set of the available VOIs for the current atlas dataset on view may be obtained from the server. VOI slice requests are also queued for the slicer, and the 2D ROI cross-section(s) for the current knife position are returned in the form of an ordered set of cartesian coordinates. More specifically, individual ROIs are returned to the client as pointlist 3D objects where all points lie in a single plane.

Other operations

Several other operations are available. For example, the client can specify a magnification or zoom level for the returned slice images, currently spanning the range from 0.1× to 10×. To support dynamic response to knife movement at higher zoom, the view is cropped and the user can pan the cropping window across the entire extent of the data set. Server throughput and latency (ping) operations provide a means for a researcher to assess the performance of their network connection from any point on the Internet.