ImageJ2: ImageJ for the next generation of scientific image data

Rueden, Curtis T.; Schindelin, Johannes; Hiner, Mark C.; DeZonia, Barry E.; Walter, Alison E.; Arena, Ellen T.; Eliceiri, Kevin W.

doi:10.1186/s12859-017-1934-z

Software
Open access
Published: 29 November 2017

ImageJ2: ImageJ for the next generation of scientific image data

Curtis T. Rueden¹,
Johannes Schindelin^1,2,
Mark C. Hiner¹,
Barry E. DeZonia¹,
Alison E. Walter^1,2,
Ellen T. Arena^1,2 &
…
Kevin W. Eliceiri ORCID: orcid.org/0000-0001-8678-670X^1,2

BMC Bioinformatics volume 18, Article number: 529 (2017) Cite this article

70k Accesses
3709 Citations
125 Altmetric
Metrics details

Abstract

Background

ImageJ is an image analysis program extensively used in the biological sciences and beyond. Due to its ease of use, recordable macro language, and extensible plug-in architecture, ImageJ enjoys contributions from non-programmers, amateur programmers, and professional developers alike. Enabling such a diversity of contributors has resulted in a large community that spans the biological and physical sciences. However, a rapidly growing user base, diverging plugin suites, and technical limitations have revealed a clear need for a concerted software engineering effort to support emerging imaging paradigms, to ensure the software’s ability to handle the requirements of modern science.

Results

We rewrote the entire ImageJ codebase, engineering a redesigned plugin mechanism intended to facilitate extensibility at every level, with the goal of creating a more powerful tool that continues to serve the existing community while addressing a wider range of scientific requirements. This next-generation ImageJ, called “ImageJ2” in places where the distinction matters, provides a host of new functionality. It separates concerns, fully decoupling the data model from the user interface. It emphasizes integration with external applications to maximize interoperability. Its robust new plugin framework allows everything from image formats, to scripting languages, to visualization to be extended by the community. The redesigned data model supports arbitrarily large, N-dimensional datasets, which are increasingly common in modern image acquisition. Despite the scope of these changes, backwards compatibility is maintained such that this new functionality can be seamlessly integrated with the classic ImageJ interface, allowing users and developers to migrate to these new methods at their own pace.

Conclusions

Scientific imaging benefits from open-source programs that advance new method development and deployment to a diverse audience. ImageJ has continuously evolved with this idea in mind; however, new and emerging scientific requirements have posed corresponding challenges for ImageJ’s development. The described improvements provide a framework engineered for flexibility, intended to support these requirements as well as accommodate future needs. Future efforts will focus on implementing new algorithms in this framework and expanding collaborations with other popular scientific software suites.

Background

ImageJ [1] is a powerful, oft-referenced platform for image processing, developed by Wayne Rasband at the National Institutes of Health (NIH). Since its initial release in 1997, ImageJ has proven paramount in many scientific endeavors and projects, particularly those within the life sciences [2]. Over the past twenty years, the program has evolved far beyond its originally intended scope. After such an extended period of sustained growth, any software project benefits from a subsequent period of scrutiny and refactoring; ImageJ is no exception. Such restructuring helps the program to remain accessible to newcomers, powerful enough for experts, and relevant to the demands of its ever-growing community. As such, we have developed ImageJ2: a total redesign of the previous incarnation (hereafter “ImageJ 1.x”), which builds on the original’s successful qualities while improving its core architecture to encompass the scientific demands of the decades to come. Key motivations for the development of ImageJ2 include:

1.
Supporting the next generation of image data. Over time, the infrastructure of image acquisition has grown in sophistication and complexity. For example, in the field of microscopy we were once limited to single image planes. However, with modern techniques we can record much more information: physical location in time and space (X, Y, Z, time), lifetime histograms across a range of spectral emission channels, polarization state of light, phase and frequency, angles of rotation (e.g., in light sheet fluorescence microscopy), and high-throughput screens, just to name a few. The ImageJ infrastructure needed improvement to work effectively with these new modes of image data.
2.
Enabling new software collaborations. The field of software engineering has seen an explosion of available development tools and infrastructure, and it is no longer realistic to expect a single standalone application to remain universally relevant. We wanted to improve ImageJ’s modularity to facilitate its use as a software library, the creation of additional user interfaces, and integration and interoperability with external software suites.
3.
Broadening the ImageJ community. Though initially developed for the life sciences, ImageJ is used in various other scientific disciplines as well. It has the potential to be a powerful tool for any field that benefits from image visualization, processing, and analysis: earth sciences, astronomy, fluid dynamics, computer vision, signal processing, etc. We wanted to enhance ImageJ’s impact in the greater scientific community by adopting software engineering best practices, generalizing the codebase, and providing unified, comprehensive, consistently structured, community-editable online resources.

From these motivations emerge the six pillars of the ImageJ2 mission statement:

Design the next generation of ImageJ, driven by the needs of the community.
Collaborate across organizations, fostering open development through sharing and reuse.
Broaden ImageJ’s usefulness and relevance across many disciplines of the scientific community.
Maintain backwards compatibility with existing ImageJ functionality.
Unify online resources to a central location for the ImageJ community.
Lead ImageJ development with a clear vision.

It is important to stress that this mission is, and always will be, informed by pragmatism. For instance, much of ImageJ’s existing user community is centered in the biosciences and related life science fields, and the core ImageJ developers and contributors are part of bioimaging laboratories as principal investigators, staff, students, consultants, etc. [3]. As such, ImageJ’s current development directions tend toward addressing problems in bioimaging. However, most image processing algorithms are generally applicable, and there are users of ImageJ in other scientific fields as well. Hence, we wish to avoid pigeonholing the software as a tool for bioimage analysis only, which would implicitly preclude it from being adopted for other purposes. One of our explicit goals is to exploit commonality across scientific disciplines, leaving the door open for others to collaborate and improve ImageJ in cases where doing so is useful.

Why ImageJ?

Any time a development effort of this scale is undertaken on an existing tool, it is worth evaluating its impact and the decision to invest such resources. The bioimage informatics field [4] is fortunate to have a wide range of software tools available in both commercial and open source arenas [5]. Open-source tools are especially important in science due to their transparency and inherent ability for sharing and extensibility [6]. This need and ability for method sharing has resulted in a plethora of open-source solutions in bioimage informatics, ranging from image acquisition tools such as μManager [7, 8]; databases such as Bio-Image Semantic Query User Environment (BisQue) [9] and OME Remote Objects (OMERO) [10]; image analysis suites such as Icy [11] and BioImageXD [12]; scientific workflow and pipeline tools such as CellProfiler [13, 14], KoNstanz Information MinEr (KNIME) [15, 16] and Pipeline Pilot [17]; and (3D) rendering applications such as FluoRender [18] and Vaa3D [19]. There are many other open, bioimaging-oriented software packages besides these, including solutions written in powerful scripting platforms such as R, Python and MATLAB. With such an extensive array of tools, does it make sense to invest in an updated ImageJ platform, rather than building on some combination of more recent tools?

The ImageJ2 project aims to do both, by rearchitecting ImageJ as a shared platform for integration and interoperability across many bioimaging software packages. ImageJ has a unique niche in that it is not a monolithic or single-purpose application, but rather a platform for discovery where the bench biologist can adapt and deploy new image analysis methods. Historically, ImageJ 1.x has been popular due to not only pre-designed tools developed for a single purpose and regularly maintained and updated, but also its powerful yet approachable plugin and macro environments that have enabled hundreds of groups to generate results through the development of thousands of customized plugins and scripts [2, 20, 21]. It is this ability for sharing, and the desire to engage the professional and amateur developer alike, that drove the development for ImageJ2. The new version of ImageJ is a platform for extensibility and cross-application cooperation, broadening the scope of ImageJ into a new effort called SciJava [22]: a collaboration of projects striving to cooperate and build on one another both socially and technically. It is our intent that with the developments detailed in this paper, the synergy between these tools, which include ImageJ, KNIME, CellProfiler, OMERO and others, will only increase as each tool continues to evolve along with current avenues of scientific inquiry, benefiting not only existing users, but new users and communities as well. See Table 1 in the “Results and discussion” section for a detailed breakdown of software that has been successfully integrated with ImageJ.

Table 1 ImageJ software integrations

Full size table

Design goals

The central technical design goals of ImageJ2 can be divided into seven key categories: functionality, extensibility, reproducibility, usability, performance, compatibility and community. In this section, we discuss the goals of ImageJ2 from its outset; for how these goals have been met in practice, see the subsequent sections.

Functionality

The overriding principle of ImageJ2 is to create powerful software, capable of meeting the expanding requirements of an ever-more-complex landscape of scientific image processing and analysis for the foreseeable future. As such, ImageJ needs to be more than a desktop application: it must be a modular, multi-layered set of functions with each layer encapsulated and building upon lower layers. In computer science terminology, ImageJ2 strives to have a proper separation of concerns between data model and display thereof, enabling use within a wide variety of scenarios, such as headless operation—i.e., running remotely on a server, cluster or cloud without a graphical user interface (UI).

At its core, ImageJ2 aims to provide robust support for N-dimensional image data, to support domains with dimensions beyond time and space. Examples include: multispectral and hyperspectral images, fluorescence lifetime measured in the time or frequency domains, multi-angle data from acquisition modalities such as light sheet fluorescence microscopy, multi-position data from High Content Screens, and experiments using polarized light. In general, the design must be robust enough to express any newly emerging modalities within its infrastructure.

Finally, it is not sufficient to provide a modular framework—ImageJ2 must also provide built-in routines as default behavior for standard tasks in image processing and analysis. These core plugins must span a wealth of algorithms for image processing and analysis, image visualization, and image file import and export. Such built-in features ensure users have an application they can apply out-of-the-box.

Extensibility

According to a survey of ImageJ users, the greatest strength of ImageJ is its extensibility [23]. From its inception [1], ImageJ 1.x has had a mechanism by which users can develop their own plugins and macros to extend its capabilities. Two decades later, a plethora of such plugins and macros have been shared and published [20]. It is paramount that ImageJ2 maintains this ease of modification and extension by its user community, and furthermore leverages its improved separation of concerns to actually make user extension easier and more powerful; e.g., if image processing plugins are agnostic to user interface, new interfaces can be developed without a loss of functionality.

A related preeminent concern is interoperability. There is no silver bullet in image processing. No matter how powerful ImageJ becomes or how many extensions exist, there will always be powerful and useful alternative tools available. Users benefit most when information can easily be exchanged between such tools. One of ImageJ2’s primary motivations is to enable usage of ImageJ code from other applications and toolkits, and vice versa, and to support open standards for data storage and exchange.

Reproducibility

For ImageJ to be truly useful to the scientific community, it must be not only technically feasible to extend, but also socially feasible, without legal obstacles or other restrictions preventing the free exchange of scientific ideas. To that end, ImageJ must be not only open source, but offer full reproducibility, following an open development process which we believe is an optimal fit for open scientific inquiry [24]. We want to enable the community to not just use ImageJ, but also to build upon it, with all project resources—revision history, project roadmap, community contribution process, etc.—publicly accessible, and development discussions taking place in public, archived communication channels so that interested parties can remain informed of and contribute to the project’s future directions. Such transparency also facilitates sensible, defensible software development processes and fosters responsibility amongst those involved in the ImageJ project. In particular, the project must be well covered by automated tests, to validate that it produces consistent results on reference data sets.

Usability

Modular systems composed of many components often have a corresponding increase in conceptual complexity, making them harder to understand and use. To avoid this pitfall, ImageJ2 employs the idea of complexity minimization: seeking sensible defaults that make simple things easy, but difficult things still possible. The lowest-level software layers should define the program’s full power, while each subsequent layer reduces visible complexity by choosing default parameters suitable for common tasks. The highest levels should provide users with the simplicity of a “big green button,” performing the most commonly desired tasks with ease—the powerful inner machinery remaining unseen, yet accessible when needed.

To bridge the gap between extensibility and usability, there must be a painless process of installing new functionality: a built-in, configurable automatic update mechanism to manage extensions and keep the software up-to-date. This update mechanism must be scalable and distributed, such that software developers can publish their own extensions on their own websites, without needing to obtain permission from a central authority.

Performance

N-dimensional images and the ever-expanding size of datasets increase the computation requirements placed on analysis routines. For ImageJ2 to succeed, it must accomplish its goals without negatively impacting performance efficiency in time—e.g., Central Processing Unit (CPU) and Graphics Processing Unit (GPU)—or space—e.g., Random-Access Memory (RAM) and disk. Furthermore, to ensure ImageJ2 meets performance needs for a wide variety of use cases, it should offer choices surrounding usage of available resources, as well as sensible defaults for balancing performance in common scenarios.

Another key consideration for performance is scalability: ImageJ must be capable of operating on increasingly huge datasets. In cloud computing, this requirement is often met via elasticity: the ability to transparently provision additional computing resources—i.e., throw more computers at the problem [25]. We are at the dawn of the “Big Data” era of computing, where both computation and storage are scalable resources which can be purchased from remote server farms. Software like ImageJ which hopes to remain effective for serious scientific inquiry into the coming decades must be architected so that its algorithms scale well to increasingly large data processed in parallel across increasingly large numbers of CPU and GPU cores.

Compatibility

There are a vast number of existing extensions—plugins, macros, and scripts—for the original ImageJ 1.x application which have proven extremely useful to the user community [20]. ImageJ2 must continue to support these extensions as faithfully as possible, while also providing a clear incremental migration path to take advantage of the new framework.

Community

The principal non-technical goal of ImageJ2 is to serve the ImageJ community as it evolves and grows; to that end, several community-oriented technical goals naturally follow. The ImageJ project must provide unified online resources including a central community-editable website, discussion forum, and online technical resources for managing community extensions of ImageJ. And the ImageJ application itself must work in concert with these resources—e.g., users should be able to report bugs directly to online issue tracking systems when something goes wrong.

Implementation

Broadly speaking, ImageJ2 components are classified into one of four domains:

SciJava. The most fundamental layers of ImageJ2 are independent from image processing, but rather provide needed functionality common to many applications. On a technical level, the SciJava core components are a set of standard Java libraries for managing extensible applications. Socially, the SciJava initiative is a pledge among cooperating organizations to work together, reuse code, and synergize wherever possible [26].
ImgLib2. To ensure generality of image analysis, ImageJ2 is built on the flexible ImgLib2 container model [27]. Decoupling the elements of image representation, ImgLib2 components enable general image processing algorithms to be reused, regardless of image type, source, or organization.
SCientific Image Format Input and Output (SCIFIO). SCIFIO components define standards for reading, writing, and translating between image formats [28]. These libraries ensure a broad spectrum of image data can be interpreted in a consistent manner across all SciJava applications.
ImageJ. Low-level components establish image metadata and algorithm patterns, built on the SciJava and ImgLib2 layers. High-level components focus on “end user” tools for working with image data, and include user interfaces, user-facing commands, and the top-level ImageJ application [29].

These layers, taken as a whole, form the ImageJ software stack [30], the core set of components upon which ImageJ-based applications are built.

Each domain is itself divided into many individual libraries, each of which targets a particular function. This separation of concerns provides a logical organization which allows targeted reuse and extension of any given functionality of interest.

The following sections describe, in order from lowest to highest level, the essential backbone libraries of ImageJ2. Note that this is not an exhaustive list of components, as many components across these domains provide secondary functions—e.g.: script languages, build management, UI elements, or targeted implementations of specific features.

SciJava common

The ground floor of the ImageJ software stack is the SciJava Common library [31], providing the core framework for creating extensible applications. The heart of SciJava Common is its application container, the Context class. Each Context encapsulates runtime application state: available extensions, open images and documents, user settings, etc. The application container paradigm allows multiple independently configured instances of SciJava applications to run concurrently within the same Java Virtual Machine (JVM).

Service framework

The application container consists of a collection of services, which are initialized dynamically at runtime. These services provide methods which operate on the system in various ways, such as opening data, manipulating images, or displaying user interface elements on screen. Taken as a whole, these service methods constitute the bulk of the Application Programming Interface (API) of ImageJ. Software developers are free to extend the system with new, needed services and/or override any aspect of behavior provided by existing services. This approach is in contrast to the most common naive design of many software projects, which use global “static” state and functions, whose behavior is difficult or impossible to override or enhance in downstream code.

The SciJava Common library itself provides the most fundamental of these services, such as:

A plugin service, which dynamically discovers available plugins using an index generated at compile time by a Java annotation processor. This plugin index is used to bootstrap the application context, since services are themselves a type of plugin.
An event service, which provides a hierarchical publish/subscribe model for event handling.
A log service, for environment-agnostic data logging.
An object service, which keeps a central typed index of available objects.
A thread service, which manages a thread pool and dispatch thread(s) for code execution.
An Input/Output (I/O) service, for reading and writing of data.
A preference service, for saving and restoring user-specific preferences.

In principle, SciJava Common is similar to frameworks such as Spring [32], offering standard software engineering patterns such as dependency injection (DI) [33] and inversion of control (IoC) [34], but tailored to the needs of collaborative scientific projects like ImageJ. For example, SciJava Common provides a generalized I/O mechanism for opening data from any source, but the library itself has no specific knowledge of how to open eXtensible Markup Language (XML) documents, microscopy image formats, or spreadsheets of numerical results—such functionality is provided by downstream components as SciJava plugins (see next section).

Plugin framework

SciJava Common provides a unified mechanism for defining plugins: extensions which add new features or behavior to the software, and/or modify existing behavior. Plugins are discovered by the system at runtime, and ordered according to assigned priorities and types, forming type hierarchies: structural trees that define how each individual plugin fits into the system. The typical pattern for a desired sort of functionality is to define a dedicated plugin type, then implement plugins that fulfill that operation in various ways. SciJava Common is designed to make virtually any aspect of an application extensible. Some of the most critical plugin categories and types include:

Core extensibility

Service – A collection of related functionality, expressed as an API. SciJava services are singletons with respect to each application context. For example, each instance of ImageJ2 has exactly one AnimationService responsible for managing animations, with methods to start and stop animations, select the dimension over which to animate, adjust frame rate, and other options. Note that while the behavior of services can certainly be modified by extensions, doing so is primarily the domain of advanced developers looking to radically alter behavior of the system.
IOPlugin – A plugin that reads data from and/or writes data to a location, such as a file on disk. For example, the SciJava layer provides I/O plugins for common text formats such as Markdown [35], while the SCIFIO layer provides an I/O plugin for image formats.

Modules

Command – An operation, more generally known as a SciJava module, with typed inputs and outputs. These modules typically appear in the menu system of the application’s user interface, but can be exposed via interoperability mechanisms in many other ways, such as nodes in KNIME or modules in CellProfiler [13]. When ImageJ users talk about “writing a plugin” they usually mean a Command. See “Module framework” below for more on SciJava modules.
ScriptLanguage – A programming language for SciJava scripts. Each script language plugin provides the logic necessary to execute scripts written in that language (e.g., JavaScript or Python) as SciJava modules with typed inputs and outputs, in a similar way to commands. It also makes it possible to express operations as code snippets that can be reused in scripts to repeat those operations.
Converter – A plugin which transforms data from one type of object to a different type of object. Converters greatly extend the concept of type conversion from what Java provides out of the box to provide automatic conversion in a wide and extensible set of circumstances. For example, it becomes possible for an algorithm to accept a string in place of a floating point numerical value, as long as that string can be parsed to such a value—or to transparently convert between normally-incompatible image data structures from different image processing ecosystems.
ModulePreprocessor – A “meta-module” which prepares modules to run. For example, the LoadInputsPreprocessor populates a module’s inputs with the last-used values as defaults, which can save the user a lot of time. Preprocessor plugins are executed in priority order as part of the module “preprocessing chain” before the module is actually executed.
ModulePostprocessor – A “meta-module” which does something with a module after it has run. For example, the DisplayPostprocessor takes care of displaying the outputs of a module after it has completed execution. Postprocessor plugins are executed in priority order as part of the module “postprocessing chain” after the module is actually executed.

User interface

UserInterface – A plugin providing an application UI. These plugins include functionality for creating and showing windows and dialogs. ImageJ2 includes a user interface written in Java’s Swing toolkit which is modeled closely after the ImageJ 1.x design, as well as a UserInterface plugin that wraps ImageJ 1.x itself. But other UIs are equally possible; since a UI is simply a type of plugin, anyone can develop their own SciJava UI without any code changes to the core system. The system is even flexible enough to display multiple UIs simultaneously.
Platform – A plugin which enables customization of behavior based on machine-specific criteria, such as specific flavor of operating system or Java language, including type, architecture, or version. For example, on Macintosh Operating System (macOS), the menu bar appears at the top of the screen, with the About, Preferences, and Quit commands relocated to the Application menu.
InputWidget – A user interface element for harvesting typed inputs. Typically, these widgets are presented as part of a form in a dialog box which prompts the user to fill in input values of a module. In principle, the widgets can be used for anything requiring typed input from the user. For example, a FileWidget allows the user to select a file (java.io.File) on disk, while a ToggleWidget provides a boolean toggle (typically rendered as a checkbox). The SciJava layer provides UI-agnostic interfaces to the common widget types, along with widget implementations corresponding to each supported UserInterface plugin. However, an extension to the system can not only implement its own data structure classes which it uses as inputs to its modules; it can also provide corresponding widgets for those structures, allowing the user to populate them from the user interface in innovative ways.
Display – A plugin for visualizing data. For example, an ImageJ2 ImageDisplay can show two-dimensional planes of N-dimensional image data in a window with sliders for controlling which plane is visible. However, the framework imposes no limits on the sorts of objects that can be visualized; other examples include the TextDisplay, which shows strings, and the TableDisplay, which shows tabular data as a spreadsheet. These plugins are typically used to display a module’s typed outputs (i.e., its results).
Tool – A collection of rules binding user input (e.g., keyboard and mouse events) to display and data manipulation actions. For example, ImageJ2’s PanTool pans a display when the mouse is dragged or arrow key is pressed; the PencilTool draws hard lines on the data within an image display. Many user interfaces render them as icons in the application toolbar.
ConsoleArgument – A plugin that handles arguments passed to the application as command line parameters. This plugin type makes the application’s command line parameter handling extensible—a feature especially important for headless operation sans user interface.

This encapsulation of functionality, coupled with a plugin prioritization mechanic, allows SciJava-based software to be fully customized or extended at any point. An application such as ImageJ is then simply a collection of plugins and services built on top of the SciJava Common framework. For instance, the ImageJ Common [36] component introduces new services specifically for opening and displaying images, specializing the functions defined in the lower-level components. Assigning these specialized functions a higher plugin priority creates a natural, flexible ordering of operations. Given that everything from user interfaces to file formats uses the SciJava plugin mechanism, the path for overriding any behavior is clear and consistent.

Finally, to keep the plugin development process as simple as possible, great care is taken throughout the codebase to adhere to interface-driven design with default method implementations whenever possible. This strategy minimizes the amount of code developers are responsible for writing, lowering the barrier to entry for creating and modifying plugins.

Module framework

To successfully interoperate with other scientific software, ImageJ algorithms must be decoupled from the various user interfaces and applications which might want to expose them to end users.

The key concept SciJava employs is that of parameterized modules: executable routines with declared input and output parameters of specific types. These modules can take the form of Command plugins or be expressed as scripts written in any supported scripting language (via available ScriptLanguage plugins; see “Plugin framework” above). For example, a user might write the following parameterized Groovy [37] script:

This script accepts two parameters as input—a name and an age—and outputs a greeting based on the input values. Note the typing: the name can be any string of characters, but the age must be an integer value; the greeting is also a string of characters. Note also that this script makes no assumptions about user interface; it is the responsibility of the framework to: A) prompt the user for the input values in the most appropriate way, B) execute the module code itself, and finally, C) process and/or display the output values in the most appropriate way.

As such, this scheme has great potential for reuse across a wide variety of contexts. For example, when running the above script from the ImageJ user interface, a Swing dialog box will pop up allowing the user to enter the name and age values; and after OK is pressed, the greeting will be displayed in a new window. However, when running the script headless from the command line interface, the input values can be passed as command line arguments and the output values echoed to the standard output stream. See Additional file 1: Figure S1 for an illustration. Since many computational tools have this concept of parameterized modules, SciJava developers need only create some general adapter code to integrate the SciJava module framework with a given tool—and then all SciJava modules become automatically available within that tool’s paradigm. We have already implemented such integration for several other tools in the SciJava ecosystem, including CellProfiler, KNIME [16], and the OMERO image server [10].

SciJava Common has an important mechanism which facilitates the extensible and configurable execution of modules: module pre- and post-processing. Developers can write ModulePreprocessor and ModulePostprocessor plugins to extend what happens when a module is executed (see “Plugin framework” above). Moreover, there are also two plugin types built on this module processing mechanism which make it easy to customize and extend how modules behave:

1.
The process of collecting module inputs is known as input harvesting. The InputWidget plugin type lets developers create widgets to harvest specific types of inputs from the user. In particular, the SciJava project provides Swing widgets for several data types (Additional file 1: Table S1).

Some inputs are also automatically populated via ModulePreprocessor code. For example, when a single image parameter is declared, an “active image preprocessor” detects the situation, populating the value with the currently active image. In this way, the user does not have to explicitly select an image upon which to operate in the common case, but the module still has semantic knowledge that an image is one of the routine’s input parameters.
2.
The process of dealing with outputs after a module executes is known as displaying. The Display plugin type lets developers visualize specific types of outputs in appropriate ways. The SciJava layer provides a basic display plugin for text outputs, which shows the text in a dedicated window, while the ImageJ layer provides additional similar display plugins for image and tabular data.

One final SciJava subsystem of note is the conversion framework, which provides a general way of transforming data from one type to another. The Converter plugin type lets developers extend SciJava’s conversion capabilities, allowing objects of one type to be used as module inputs of a different type, in cases where the two types are conceptually analogous. For example, data stored in memory as a MATrix LABoratory (MATLAB) matrix can be expressed as an ImageJ image object, even though MATLAB matrices are not natively ImageJ images [38]. When a suitable converter plugin is present, modules capable of operating only on MATLAB matrices become transparently capable of accepting ImageJ images as inputs, thanks to the framework’s auto-conversion. Similarly, a converter between ImageJ and the Insight ToolKit (ITK) [39] images greatly streamlines use of ITK-based algorithms within ImageJ [40].

ImageJ common

Meeting the needs of contemporary scientific image analysis requires a flexible and extensible data model, including support for arbitrary dimensions, data types and image sizes. To this end, we have chosen to model ImageJ2 images using the ImgLib2 library, which itself provides an extensible, interface-driven design that supports numeric (8-bit unsigned integer, 32-bit floating point, etc.) and non-numeric data types. It also provides great flexibility regarding the source and structure of data. Out of the box, ImgLib2 provides several data sources and sample organizations, including use of a single primitive array (“array image”), one array per plane (“planar image”), and block-based “cell image.” However, the library remains general enough that alternative structures are also feasible. To quote the ImgLib2 article [27]:

The core paradigm [of ImgLib2] is a clean separation of pixel algebra (how sample values are manipulated), data access (how sample coordinates are traversed), and data representation (how the samples are stored, laid out in memory, or paged to disc). ImgLib2 relies on virtual access to both sample values and coordinates, facilitating parallelizability and extensibility.

ImageJ Common provides a unification of the type and storage-independence of ImgLib2 with the SciJava Common plugin framework (described above). A Dataset interface provides the fundamental representation of ImageJ images, collections of images, and corresponding metadata: (ROIs), visualization settings, sample coordinates and physical calibrations, and much more. Also provided are plugins and services for working with these Dataset objects. Together, these classes form the access points for higher-level components to open, save, generate and process these images.

Note that as of this writing, elements of the ImageJ Common data model and corresponding services are still stabilizing. As such, we do not describe these structures in technical detail here.

SCIFIO

An essential goal of ImageJ2 is to establish universal image analysis routines, with no limits on application; however, the proliferation of proprietary image formats from scientific instruments creates a major obstacle to this ambition. To overcome this issue, the SCIFIO core library establishes a common framework for reading, writing and translating image data to and from the ImageJ Common data model, as well as between domain-specific standard metadata models. SCIFIO builds on the services provided in SciJava Common and ImageJ Common, defining image Format and metadata Translator plugin types to encapsulate the operations necessary to take an image source and standardize it as an ImageJ Dataset.

SCIFIO builds upon SciJava Common’s core I/O infrastructure, which allows it to operate on most data locations independent of their nature. SciJava Common provides a Location interface which acts as a data descriptor, similar to a Uniform Resource Identifier (URI). This Location interface is specialized according to the nature of the data; for example, a URLLocation identifies data served by a remote Uniform Resource Locator (URL), while an OMEROLocation (part of the ImageJ-OMERO integration [41]) identifies an image from an OMERO server. For data locations whose raw bytes can be accessed randomly and/or sequentially (e.g., remote URLs, but not OMERO images), SciJava Common provides a DataHandle plugin type which enables such access. The core library provides DataHandle plugins for several kinds of data locations, including files on disk, remote URLs, and arrays of bytes in local computer memory. Developers can easily create new DataHandle plugins which provide random access into additional sorts of locations, and SCIFIO will be able to use them transparently without any changes to existing Format or Translator plugins.

The Format plugin API is architected to support reading and writing of image data in chunks, which provides scalability. It is no longer necessary to have a large quantity of computer RAM to work with large images—SCIFIO reads the data from the source location on demand, paging it into and out of memory as needed. SCIFIO’s caching mechanism persists any changes made to image pixels, even when chunks leave memory, by using temporary storage on disk.

SCIFIO Translator plugins provide the means to translate not only between image formats, but between common metadata models of various scientific disciplines. For example, the Open Microscopy Environment (OME) defines a data model called OME-XML [42], for which the SCIFIO-OME-XML component provides a suite of translators to and from ImageJ Common data structures. In this way, SCIFIO has the potential to bridge interoperability gaps across various discipline-specific scientific software packages.

Further details about SCIFIO can be found in the BioMed Central (BMC) Bioinformatics software article “SCIFIO: an extensible framework to support scientific image formats” [28].

ImageJ ops

ImageJ’s ultimate purpose is image processing and analysis. To that end, we have crafted the ImageJ Ops component: ImageJ2’s shared, extensible library of reusable image processing operations. As of version 0.33.0, the core Ops library provides 788 Op plugins across nearly 350 types of ops in more than 20 namespaces, covering functionality such as: image arithmetic, trigonometry, Fourier transformations, deconvolution, global and local thresholding, image statistics, image filtering, binary morphological operations, type conversion, image transformations (scaling, rotation, etc.)—even 2- and 3-dimensional geometric operations such as marching cubes 3D mesh generation (see Fig. 1 for examples). A thorough treatment of available ops can be found in the ImageJ Ops tutorial notebook [43].

ImageJ Ops was conceived with three major design goals: 1) easy to use and extend; 2) powerful and general; and 3) high performance. To achieve all three of these goals, Ops utilizes a plugin-based design enabling “extensible case logic.” Ops defines a new plugin type, Op, each of which has a name and a list of typed parameters, analogous to a function definition in most programming languages. When invoking an op, callers typically do not specify the exact Op plugin to use, but instead specify the operation’s name and arguments; the Ops framework then performs a matching process, finding the optimal fit for the given request. For example, calling math.add with a planar image and a 64-bit floating point number leads to a match of net.imagej.ops.math.ConstantToPlanar Image.AddDouble, which adds a constant value to each element of an image, whereas calling math.add with two planar images results in a match of net.imagej.ops.math.IIToIIOutputII.Add, which adds two images element-wise.

This scheme is similar to—but more powerful than—the method overloading capabilities of many programming languages: op behavior can be further specialized by tailoring Op implementations for specific subclasses, generic parameters, and Converter substitutions (see “SciJava common” above). Consider an op sqrt(image), which computes the element-wise square root of an image. If we implement this op as sqrt(Dataset), we miss out on performance optimizations for ArrayImg, an ImgLib2 container type where the entire collection of image samples is stored in a single Java primitive array. However, if we only implement sqrt(ArrayImg), we are restricted in supported data types, since not all images can be stored in such a manner. The power of the Ops matching approach is that both of these and more can coexist simultaneously and extensibly, and the most specific will always be selected at runtime.

Furthermore, as algorithm implementations increasingly become available for the GPU via libraries such as Open Computing Language (OpenCL) [44] and Nvidia’s Compute Unified Device Architecture (CUDA) [45], as well as for clusters via libraries such as Apache Spark [46], such implementations could also be expressed as ops so that they can be selected automatically based on the currently available hardware environment.

The Op plugin type extends SciJava’s Command, and therefore all ops are SciJava parameterized modules, usable anywhere SciJava modules are supported—see the “Module framework” section in “SciJava common” above. Like standard modules, an op declares typed inputs and outputs. However, unlike modules in general, an op must be a “pure function” with a fixed number of parameters and no side effects; i.e., it must be deterministic in its behavior, operating only on the given inputs, and populating or mutating only the given outputs. These restrictions provide some very useful guarantees which allow the system to reason about an op’s use and behavior; e.g., after computing an op with particular arguments once, the result can be cached to dramatically improve subsequent time performance at the potential expense of additional space. Properly constructed ops will also always be usable headless because they do not rely on the existence of UI elements.

Op chaining and special ops It is often the case in image processing that an algorithm can be expressed as a composition of lower level algorithms. For example, a simple difference of Gaussians (“DoG”) operation is merely two Gaussian blur operations along with a subtraction:

dog(image,σ ₁,σ ₂)=sub(gauss(image,σ ₁),gauss(image,σ ₂))

For users calling into the Ops framework via scripting, the core library provides an eval op backed by SciJava’s expression parser library, which enables executing such sequences of ops via standard mathematical expressions, including use of unary and infix binary operators.

For developers, the Ops library provides a mechanism for efficient chaining of ops calls. An op may declare other ops as inputs, resulting in a tree of ops which are resolved when an op is matched; the matched op instance can then be reused across any number of input values. In this way, very general operations can be created to address a broad range of use cases—e.g., the map operation provides a unified way of executing an op such as math.sqrt(number) element-wise on a collection (e.g., an image) whose elements are numbers. Indeed, in the case of DoG, the Ops library’s baseline implementation takes an image as input, along with two filter.gauss ops and a math.sub op, and then invokes them on the input image. The baseline stats.mean implementation is similar, built on the stats.sum, stats.size and math.div ops. Higher level DoG ops provide sensible defaults, enabling calls like dog(image, sigma1, sigma2) to work, making common operations simple, while leaving the door open for additional customization as needed.

To facilitate type-safe and efficient chaining of ops, the Ops library has a subsystem known as special ops. Such special ops are specifically intended to be called repeatedly from other ops, without needing to invoke the op matching algorithm every time. This repeat usage is achieved in a type-safe and efficient way by explicitly declaring the types of the op’s primary inputs—i.e., the inputs whose values can be efficiently varied across invocations of the op—as well as the type of the op’s primary output.

Special ops have two major characteristics beyond regular ops. First, each special op has a declared arity, indicating the number of primary inputs, which are explicitly typed via Java generics and can thus efficiently vary across invocations of the op. Three arities are currently implemented: nullary for no inputs, unary for one input, and binary for two inputs. It is important to note that unlike a formal mathematical function, a unary special op may have more than one input parameter—the “unary” in this case refers to the number of explicitly typed parameters intended to vary when calling an instance of the op multiple times. For instance, in the DoG example above, the baseline DoG is declared as a unary op, so that the input image can vary efficiently while the sigmas etc. are held constant in value.

Secondly, every special op is one of three kinds:

A function operates on inputs, producing outputs, in a way consistent with the functional programming paradigm. Inputs are immutable, and outputs are generated during computation and subsequently also immutable. Functions are very useful for parallel processing since they are fully thread-safe even when object references overlap—but this safety comes at the expense of space, since they offer no way to reuse preallocated output buffers.
A computer is similar to a function, but populates a preallocated output object instead of generating a new object every time. Computers have many of the same advantages of functions, but provide the ability to reuse preallocated output buffers to improve efficiency in space and time.
An inplace op mutates its input(s) in place—i.e., its input and output are the same object. Inplace ops are highly space efficient, but lack the mathematical guarantees of functions and computers, since they destroy the original input data.

Some ops are implemented as hybrids, offering a choice between two or more of the function, computer and inplace computation mechanisms. Users of the ops library—even advanced users—will rarely if ever need to know about this implementation detail, but for developers crafting new ops, it is convenient to have unifying interfaces which provide common logic for combining these paradigms. See Additional file 1: Table S2 for a complete breakdown of the special op kinds and arities.

ImageJ legacy

To maximize backwards compatibility with ImageJ 1.x, ImageJ2 must continue to provide access to the complete existing UI and API with which ImageJ users are familiar, while also making all new ImageJ2 features available for exploration and use. Furthermore, to bridge the gap, ImageJ2 must provide improved functionality transparently when possible, as well as support seamless “mixing and matching” of the two respective APIs. In this way, ImageJ2 can enable gradual migration to the more powerful capabilities of ImageJ2, while empowering developers’ contributions to the framework to be immediately effective. To achieve this goal, we identified the major functional pathways of ImageJ 1.x and reworked them to delegate first to ImageJ2 equivalents, falling back on the old behavior if needed.

There are two ImageJ components dedicated to maintaining backwards compatibility with ImageJ 1.x. The lower level of the two is the IJ1-patcher: using a tool called Javassist [47] to perform an advanced Java technique known as bytecode manipulation, ImageJ 1.x code is modified at runtime to expose callback hooks at critical locations—e.g.: when opening images with File ⊳ Open…, closing the ImageJ application, or displaying UI components. These hooks are built using the SciJava plugin infrastructure, allowing new behavior to be injected into ImageJ 1.x despite the fact that it was not designed to support such extensibility. In essence, ImageJ2 “rewrites” portions of ImageJ 1.x at runtime to make integration possible. This approach is necessary because altering ImageJ 1.x directly to enable such hooks would break backwards compatibility with existing macros and plugins, ruining established scientific workflows which have otherwise remained functional across many years.

By default, these hooks are exploited to inject ImageJ2 functionality in the second compatibility layer: ImageJ Legacy. ImageJ2 intercepts an ImageJ 1.x request and attempts to delegate to its own routines. For example, in our implementation of the File ⊳ Open… hook, we use the SciJava I/O service, which provides extensible support for data types via SciJava I/O plugins. This allows the full power of SCIFIO to be called automatically by File ⊳ Open… without requiring users to select individual loader plugins. In this way, ImageJ2 exposes new “seams” which provide extensibility points not available in the standalone ImageJ 1.x project [48].

A second major function of ImageJ Legacy is to provide a wrapping legacy UI: an ImageJ2 UserInterface plugin that reuses the ImageJ 1.x UI, but maintains synchronization between respective data structures. For example, consider the ImagePlus structure in ImageJ 1.x and its equivalent, the Dataset, in ImageJ2. By default, an ImagePlus and Dataset could not be interchanged; they have different Java class hierarchies, and with ImageJ2’s expanded data model, a Dataset is more expressive than an ImagePlus. However, requiring plugins to “select one” would impose a technical barrier, even if both structures are available in the same application. Thus, the legacy UI notes when either an ImagePlus or a Dataset is created and ensures a complementary instance is mapped, via SciJava Converter plugins. This brings the ImageJ 1.x and ImageJ2 worlds closer together: when an image is opened, it can be used by plugins that would take an ImagePlus or a Dataset regardless of whether that image was opened via an ImageJ 1.x or ImageJ2 mechanism. Furthermore, because conversion is handled in the ImageJ Legacy layer, individual plugins do not require knowledge of the synchronization.

Shared image data structures are but one aspect of the legacy UI’s synchronization. Others include logging, notification, and status events—essentially all UI events are mapped across paradigms. Whenever possible, these conversions are achieved using an adapter class that implements a common interface (e.g., Dataset), which wraps the object of interest (e.g., ImagePlus) by reference. This approach enables information to be translated between ImageJ 1.x and ImageJ2 structures on demand, while minimizing the performance impact. Wrapping by reference also mitigates the burden of updates; once synchronization is established, changes to the underlying object are automatically reflected in the wrapper.

ImageJ updater

The ImageJ Updater is the mechanism by which the available and installed components of ImageJ are managed. At its core, the Updater is a flexible component for tracking ImageJ update sites: endpoints containing versioned collections of files. Users can pick and choose which update sites they wish to enable, with ImageJ’s core functionality offered on the base “ImageJ” update site, which is on by default. Distributions of ImageJ such as Fiji Is Just ImageJ (Fiji) [49] extend this base with additional functionality (Fig. 2) in the form of more plugins, scripts, sample images, color lookup tables (LUTs), etc., leveraging the ability to override ImageJ’s base behavior using SciJava’s plugin priority mechanism (see “SciJava common” above for details).

The Updater stores metadata in a db.xml.gz file in the root of each update site, which describes the files that are part of that update site, including checksums and timestamps for all previous versions. In this way, the Updater can tell whether each local file is: A) an up-to-date tracked file; B) an old version of a tracked file; C) a locally modified version of a tracked file; or D) an untracked file. Update sites are served to users over Hypertext Transfer Protocol (HTTP). Developers may upload files to an update site via an extensible set of protocols, as defined by Uploader plugins. The core ImageJ distribution includes plugins for Secure SHell (SSH), Secure CoPy (SCP), Secure File Transfer Protocol (SFTP) and Web Distributed Authoring and Versioning (WebDAV), but in principle, the Updater makes no assumptions about how files are uploaded.

The db.xml.gz structure was originally designed for use with the Fiji Updater, the ImageJ Updater’s predecessor. The logic of the Fiji Updater was migrated into the core of ImageJ2, with backwards compatibility preserved for existing Fiji installations. As part of that migration, the Updater was heavily refactored to be UI agnostic, such that additional user interface plugins for the Updater could be created which leverage the same core. Out of the box, ImageJ provides two different user interfaces for the Updater: a command-line tool intended for power users and developers, and a Swing UI intended to make updating easy for end users. When ImageJ is first launched, it automatically runs the “Up-to-date check” command, which then displays the Updater UI if updates are available from any of the currently enabled update sites.

Results and discussion

ImageJ has transformed from a single-user, single-bench application to a versatile framework of extensible, reusable operations. In the following sections, we discuss how each core aspect of ImageJ2 has impacted community usage and how we expect these qualities to shape future developments.