We developed ECoG ClusterFlow in close collaboration with neuroscientists (including neurosurgeons) to guide the design of our analysis framework and to ensure that it would be truly valuable as an exploratory tool.
Figure 1 shows the pipeline of our system. The input to our system is: 1) the processed electrical signal data originating from each electrode in the ECoG grid and, 2) its corresponding pairwise dynamic correlation network. The dynamic network data is pre-processed to derive dynamic clusters. Visualization methods, such as data aggregation, are applied to the cluster data in the pre-computation phase and final visualizations are generated.
Based on our conversations with domain experts and the network task taxonomy by Ahn et al. [43], we have identified the following domain questions of interest:
-
Identify temporal brain states (Q1): What activation patterns are consistent over a continuous period of time?
-
Identify transitions between brain states (Q2): Given the brain states, what patterns characterize their transition to another state?
-
Compare the evolution patterns associated with different brain states (Q3): What patterns underlie the brain states during normal versus diseased condition?
-
Assess changes in community membership (Q4): Given a spatial region of interest in the brain, how do the clusters belonging to these regions change over time?
These questions led us to establish the following system design goals:
-
Timeline-based visualizations (G1): Support views that display the time-varying cluster information on a static display to take advantage of the user’s visual perception instead of cognition (time-to-time mapping)
-
Multiple levels of detail and abstraction (G2): Support views that enable neuroscientists to explore the data at multiple levels of granularity for analysis
-
Holistic visualizations (G3): Support visual designs that combine multiple data attributes like cluster membership and its electrical activation
These goals are addressed in our system by two major views: the Cluster Evolution View and the Electrode View.
Cluster evolution view
The cluster evolution view (Fig. 3) highlights the salient patterns of the cluster evolution including the emergence, death, contraction, expansion, merging and splitting of clusters (Q2, Q3, Q4). Through this view, analysts can compare and analyze modular signatures (cluster evolution patterns) over time and identify important time intervals and distinct brain states. The cluster evolution patterns are represented using a flow-based visualization [21, 22] (G1) (alluvial diagram), where the clusters metaphorically flow like a river with split/merge tributaries from left to right.
Formally, at each timestep t on the horizontal axis, rectangular blocks represent clusters C
t,i
where the height of each block corresponds to the cluster’s size at that timestep. Flow-based transition links L
i,j
, where i is the source community and j is the sink community, connect clusters to show changes in the community structure over time. We model these links as Bezier curves, to generate a continuous representation of the transition between successive communities [22]. Figure 3 shows the evolution of dynamic clusters for five timesteps. Furthermore, to easily assess the community membership in dynamic clusters, we color communities using solid coloring, using N perceptually distinct colors from a qualitative colorbrewer [44] colormap.
Cluster tracking
To support the two-stage cluster detection approach, it is necessary to determine correspondences between clusters in consecutive timesteps. Based on the input from neuroscientists, we have investigated two approaches to compute this matching: (1) maximum overlap tracking and (2) computing the globally optimal match.
Maximum overlap tracking (Fig. 3
a, c) is a greedy algorithm that iteratively matches the two clusters in consecutive timesteps that share the maximum number of electrodes. This process is repeated until no overlapping clusters remain. This approach may not always produce an intuitive correspondence between clusters. For example, in Fig. 3
a and c, clusters C
1,2 and C
2,2 have maximum overlap (of 11 electrodes) and are paired in the first iteration. This only leaves C
2,1 as possible match for C
1,2 in the second iteration, even though the overlap between C
1,2 and C
2,1 is relatively small (only two electrodes).
To find the globally optimal assignment, our second approach picks the best overall match between all clusters in consecutive timesteps. We define a similarity measure
$$ sim = \frac{\left|C_{t,i} \cap C_{t+1,j}\right|}{\left|C_{t,i} \cup C_{t+1,j|}\right|} $$
between clusters C
i
and C
j
in consecutive timesteps t and t+1, similar to the approaches by Greene et al. [45] and Sallabury et al. [20]. Next, we compute a similarity matrix comprised of the pairwise similarity measures between all possible cluster combinations. To avoid matching of clusters with small overlap, we set to zero those similarity values that are below a threshold θ. To match clusters, we consider all possible cluster matchings between timesteps—by considering all possible permutations of clusters—and compute a global similarity value as the sum of the similarity values for all matched clusters. The overall best match is the permutation that maximizes global similarity. While considering all possible permutations is computationally expensive, we usually consider only a small number of clusters (approximately seven) per time step, keeping this approach tractable. Figure 3
b shows the best overall match for our example, matching clusters C
1,1 and C
2,1 as well as clusters C
1,2 and C
2,2, a more intuitive choice than the result obtained by maximum overlap tracking. Figure 3 shows an example of this approach for an artificial dataset with three dynamic communities. The two approaches differ in the community results starting at timestep four.
Sorting and ordering of nodes
To enhance the visual traceability of the clusters, the node layout of the graph should ideally minimize edge crossings with optimal ordering of the nodes (clusters) at each vertical axis. To determine such an ordering, we must take all the timesteps into consideration. Several methods have been proposed to compute such an ordering [20, 22]. Our approach handles more timesteps by not considering the individual elements contained within clusters and dividing the sorting procedure into N individual blocks of T timesteps. To reduce the computational complexity—to achieve the least start-up-time of 40–60 s and to scale to up to 60 timesteps—our heuristic solution (barycenter approach [46]) sweeps horizontally across all the clusters over N blocks of T timesteps (where NT is the total number of timesteps in the data) in a front-to-back and back-to-front manner to optimize the order and position of these communities on the vertical axis. This procedure results in a cluster ordering that minimizes the link distance between the timesteps.
K-Cluster heatmap
Cluster analysis results can be sensitive to noise and prone to overfitting [42]. The K-Cluster heat map produces important information for the evaluation of the uniqueness of the number of clusters detected per timestep. Consensus clustering uses a cumulative distribution function (CDF) to determine an appropriate number of clusters K. In most cases, the likelihood for a single value of K will be large compared to the others, and the confidence that the chosen number of clusters is correct is high. The K-Cluster heat map (Fig. 2) shows the likelihood for a range of values of K for each time step. Black denotes high likelihood and white low likelihood. Using this heat map, analysts can identify timesteps where multiple values for K are almost equally likely and where confidence in the clustering results is low.
Electrode view
The electrode view shows (Fig. 4) cluster membership and electrical activity in a spatial context (G3) (Fig. 9
a, b), enabling the user to identify important spatial cluster evolution patterns (Q1, Q2 and Q3). These evolution patterns (Fig. 5) include 1) spatial cluster distribution, i.e., a cluster originally comprised of spatially adjacent electrodes splits into disjoint parts, 2) spatial cluster combination, i.e., a cluster consisting of disjoint regions becomes spatially coherent, and 3) spatial activation, i.e., the electrical activity of electrodes increases over time. The electrode view places interactive glyphs–representing electrodes–on a 2D sagittal projection of a subject-specific reconstructed brain MRI model to provide the spatial context.
To support exploration at multiple temporal scales (G2) and effective comparison of cluster characteristics—i.e., cluster membership and electrical activity of the individual electrodes—over the spatial domain, our visualization method aggregates N multiples of continuous timesteps on each electrode view. Furthermore, interactive techniques that can be applied to glyphs, make it possible for the user to define the N multiples to be aggregated on each glyph.
The aggregation techniques provide the user with a detailed overview of the underlying data while not being overwhelmed by the entire data [47]. However, aggregated views have a drawback: they can lead to cognitive information overload. Nonetheless, for the multiples we consider, we found that our visualizations are better suited for the tasks performed by our collaborating neuroscientists.
Glyph design
All our glyph designs display n user-defined timesteps per individual electrode view, to save presentation space and to facilitate comparison between timesteps. We considered three visual designs for our glyphs, following some of the data aggregation guidelines specified by Elmqvist et al. [47]. The first design uses vertically stacked bar charts. Each bar represents one time step with its color indicating cluster membership and its height corresponding to electrical activity (Fig. 6
a). The second design uses a clock metaphor [32] and subdivides each glyph into n equal slices. Each slice—starting at the top in clockwise order—represents a time step with its color indicating cluster membership and its radius representing electrical activity (Fig. 6
b). The third design is a variation of the second design and uses slice opacity instead of radius to represent the electrical activation level (Fig. 6
c).
The first design depicts changes in cluster membership intuitively, but requires a large amount of glyph space. In the second design, the varying glyph sizes in the entire spatial layout impede the neuroscientists to assess the relative positions of the electrodes. Overall, our domain science collaborators preferred the third design where the glyph shape remains constant and only color and opacity are used to convey information. We use this design in our system and the remaining discussion in this paper is based on it.
Hierarchical exploration
Patterns in brain activity occur at different temporal scales, where the appropriate scale may not be known a priori. To facilitate discovery of these patterns and scales, our tool controls the timesteps that can be displayed in a single electrode view (Fig. 7). At the extremes, the method either displays all timesteps in a single electrode view (low granularity at the top level) or each time step in a separate electrode view (high granularity at the bottom level). Between these extremes, different levels of aggregation are possible. Different approaches exist to search for temporal evolution patterns. Top-down analysis starts with all timesteps in a single view and decreases aggregation until a pattern of interest is found. A bottom-up approach starts analysis with each time step in a separate view and increases aggregation until a pattern is found. However, exploration may also start at mid-level, in situations where the user already has a notion at what temporal scale a pattern may be identified.
Layout
To identify low-level changes in a bottom-up based approach, several low-level electrode views need to be examined simultaneously. The system presents the views in a from-left-to-right order, synchronized with the timeline. Visual scalability is problematic as screen resolution is small relative to data resolution. Therefore, ECoG Cluster Flow utilizes a space-saving layout to achieve a tight synchronization between the spatial (electrode) view and temporal attributes (cluster evolution view) of the data. Each electrode view is ordered alternatively above and below the cluster evolution view, see Fig. 8, timesteps 3–10, to achieve the desired integration. The cluster evolution view is expanded or contracted based on the combined space allocated to all electrode views. To emphasize the granularity of the electrode view, the space assigned to each electrode view is proportional to the corresponding number of time points.
We use the formula
$$ w_{i} = w_{min}\times S_{f}, S_{f} \geq 1 $$
to define the width w
i
of each electrode view, and
$$S_{f} = C \times \frac{W_{max}}{w_{min}\times max(N_{x_{1}},N_{x_{2}})} $$
to define the expansion factor, S
f
, for each electrode view, The variables \(N_{x_{1}}\) and \(N_{x_{2}}\) represent the numbers of the electrode views for the x
1− and x
2− axes, representing the “above” and “below” cluster evolution views, respectively, The value of W
max
is the maximally possible horizontal screen display width, C is a user-specified constant, and w
min
defines the minimal size of each electrode view.
Interactivity
The strength of our system is the fact that it supports fast and intuitive analysis of cluster data at interactive rates. Based on previous work [48] and the suggestions made by our collaborating neuroscientists, we selected interaction techniques satisfying the user-defined design objectives. These are the interaction principles satisfied by our system:
-
Overview first, then details on demand: The cluster evolution view is an intuitive way for users to first obtain a simple outline of the entire dataset. Overview trends and outliers can be easily captured, enabling users to quickly determine a time interval of importance, and perform detailed exploration on the visualization.
-
Focus+context: This technique [49] allows users to focus on small-scale evolution patterns while preserving the overall context. It utilizes smooth, animated changes to track the patterns in focus. Furthermore, the technique facilitates comparison of patterns across time-intervals over the temporal and spatial domains.
-
Highlighting and linking: The cluster evolution view and electrode views are coordinated using brushing and linking. Users can select timesteps of interest and observe the evolution of patterns via the cluster evolution view or the electrode view. All visual elements are supported by simple tooltips providing information about the meaning of visual encodings.