Multi-scale visual analysis of time-varying electrocorticography data via clustering of brain regions

Communities for dynamic graphs

When exploring communities in dynamic graphs, existing techniques primarily use animation (time-to-time mapping) or static timeline-based (time-to-space mapping) visualization methods to depict modular changes over time.

In animations, the community structure of the network is shown by color-coding the nodes or partitioning the drawing space into sections [9–12] or nested blocks [13] (if the data is hierarchical). Due to their reliance on short-term memory, animations increase the cognitive load during analysis [14]. One way to mitigate this problem is to maintain the ‘mental map’ of the layout by minimizing node movement in the animation [15]. An alternate approach to decrease the cognitive load is to place multiple graph representations along a timeline using small multiples [16]. However, this multi-view approach leaves the user with the manual task of assimilating and identifying changes.

To address this problem, several approaches utilize timeline-based representations [17–19], visualizing only the evolution of clusters over time. In a timeline view, each segment along the axis perpendicular to the timeline represents a cluster identified at that particular timestep. The links between two axes represent the changes in the cluster affiliation of the nodes. The arbitrary ordering of the nodes along the vertical axis may increase link crossings between axes, inhibiting easy comprehension of the evolution patterns. To address this issue, Reda et al. [18] and Sallabury et al. [20] employ sorting techniques to place active and stable communities at the top of the vertical axis.

To further support the comprehension of transitions between communities, alluvial diagrams [21] model the links between clusters in different vertical axes as split-merge ribbons [17, 22, 23]. This approach enhances the visual traceability of important cluster evolution patterns.

Reda et al. [24] visualize the evolution of time-varying clusters while taking into account the spatial context, and by linking a space-time cube with a timeline representation. In contrast, our method provides the spatial context showing a multi-scale dynamic evolution patterns in 2D space, reducing visual clutter and occlusion. Our technique matches clusters through a best overall match algorithm, enabling intuitive identification of time-varying community patterns.

Spatio-temporal data

Previous visual analysis methods for spatio-temporal data utilize either integrated or separated views [25].

Integrated views visualize spatial and temporal data in one view. Superimposing temporal graph data onto a spatial view [26] and visualizing a 3D space-time cube timeline over a 2D spatial view [27] are two examples of integrated views. Another hybrid 2.5D approach proposed by Tominski et al. [28] displays temporal information on top of a 2D spatial layout. However, for a large number of timesteps or data points, these views can easily become cluttered and occluded.

Separated views overcome visual clutter by using dedicated views to present different aspects of spatio-temporal data. Plug et al. [29] link data in spatial and temporal domains by using small multiples of maps, superimposing a subset of temporal data on each of the spatial maps. Jern et al. [30] utilize color to link spatial and temporal data. Other methods [31] for static data use interaction techniques to link data in both domains, requiring substantial and concentrated eye movements for visual analysis.

To overcome such drawbacks, visual glyph designs aggregate spatio-temporal attributes that not only reduce the size of the represented data but also enable intuitive comparison of temporal data. Glidgets [32] depict temporal changes by segmenting glyphs into time slices, enabling the comparison of attributes over time. Related work by Nan Cao et al. [33] and Erbacher et al. [34] uses glyphs that aggregate temporal data to summarize the entire dataset with the overall goal of detecting anomalous behavior in the network.

ECoG ClusterFlow uses a combination of the aforementioned concepts to provide unique glyph-based designs and visual analysis methods that show the overall modular changes of the network.

Visualization systems for ECOG brain connectivity data

Graimann et al. [35] presented methods to visualize event-related desynchronization and synchronization (ERD/ERS) patterns of implanted electrodes. Research done by Korzeniewska et al. [36] and Cristhian et al. [37] included the visualization of causal relationships among electrode sites. Kubanek et al. [38] recently presented a tool for visualizing topographies of ECoG cortical activity on a 3D model of the cortex. Although these approaches satisfactorily portray the spatial layout of the brain, they do not support the visualization of time-varying modular data for functional ECoG brain networks.

There exists a need for tools supporting efficient, high-level data analysis and exploration, including dynamic cluster analysis as a main focus. To aid the process of generating and verifying scientific hypotheses, a thorough visual understanding of the intricate spatio-temporal patterns of ECoG data is necessary. We address this need with a stand-alone application that allows a user to explore cluster community evolution at varying granularity.

Cluster evolution view

The cluster evolution view (Fig. 3) highlights the salient patterns of the cluster evolution including the emergence, death, contraction, expansion, merging and splitting of clusters (Q2, Q3, Q4). Through this view, analysts can compare and analyze modular signatures (cluster evolution patterns) over time and identify important time intervals and distinct brain states. The cluster evolution patterns are represented using a flow-based visualization [21, 22] (G1) (alluvial diagram), where the clusters metaphorically flow like a river with split/merge tributaries from left to right.

Formally, at each timestep t on the horizontal axis, rectangular blocks represent clusters C _t,i where the height of each block corresponds to the cluster’s size at that timestep. Flow-based transition links L _i,j , where i is the source community and j is the sink community, connect clusters to show changes in the community structure over time. We model these links as Bezier curves, to generate a continuous representation of the transition between successive communities [22]. Figure 3 shows the evolution of dynamic clusters for five timesteps. Furthermore, to easily assess the community membership in dynamic clusters, we color communities using solid coloring, using N perceptually distinct colors from a qualitative colorbrewer [44] colormap.

Electrode view

The electrode view shows (Fig. 4) cluster membership and electrical activity in a spatial context (G3) (Fig. 9 a, b), enabling the user to identify important spatial cluster evolution patterns (Q1, Q2 and Q3). These evolution patterns (Fig. 5) include 1) spatial cluster distribution, i.e., a cluster originally comprised of spatially adjacent electrodes splits into disjoint parts, 2) spatial cluster combination, i.e., a cluster consisting of disjoint regions becomes spatially coherent, and 3) spatial activation, i.e., the electrical activity of electrodes increases over time. The electrode view places interactive glyphs–representing electrodes–on a 2D sagittal projection of a subject-specific reconstructed brain MRI model to provide the spatial context.

To support exploration at multiple temporal scales (G2) and effective comparison of cluster characteristics—i.e., cluster membership and electrical activity of the individual electrodes—over the spatial domain, our visualization method aggregates N multiples of continuous timesteps on each electrode view. Furthermore, interactive techniques that can be applied to glyphs, make it possible for the user to define the N multiples to be aggregated on each glyph.

The aggregation techniques provide the user with a detailed overview of the underlying data while not being overwhelmed by the entire data [47]. However, aggregated views have a drawback: they can lead to cognitive information overload. Nonetheless, for the multiples we consider, we found that our visualizations are better suited for the tasks performed by our collaborating neuroscientists.

Glyph design

All our glyph designs display n user-defined timesteps per individual electrode view, to save presentation space and to facilitate comparison between timesteps. We considered three visual designs for our glyphs, following some of the data aggregation guidelines specified by Elmqvist et al. [47]. The first design uses vertically stacked bar charts. Each bar represents one time step with its color indicating cluster membership and its height corresponding to electrical activity (Fig. 6 a). The second design uses a clock metaphor [32] and subdivides each glyph into n equal slices. Each slice—starting at the top in clockwise order—represents a time step with its color indicating cluster membership and its radius representing electrical activity (Fig. 6 b). The third design is a variation of the second design and uses slice opacity instead of radius to represent the electrical activation level (Fig. 6 c).

The first design depicts changes in cluster membership intuitively, but requires a large amount of glyph space. In the second design, the varying glyph sizes in the entire spatial layout impede the neuroscientists to assess the relative positions of the electrodes. Overall, our domain science collaborators preferred the third design where the glyph shape remains constant and only color and opacity are used to convey information. We use this design in our system and the remaining discussion in this paper is based on it.

Hierarchical exploration

Patterns in brain activity occur at different temporal scales, where the appropriate scale may not be known a priori. To facilitate discovery of these patterns and scales, our tool controls the timesteps that can be displayed in a single electrode view (Fig. 7). At the extremes, the method either displays all timesteps in a single electrode view (low granularity at the top level) or each time step in a separate electrode view (high granularity at the bottom level). Between these extremes, different levels of aggregation are possible. Different approaches exist to search for temporal evolution patterns. Top-down analysis starts with all timesteps in a single view and decreases aggregation until a pattern of interest is found. A bottom-up approach starts analysis with each time step in a separate view and increases aggregation until a pattern is found. However, exploration may also start at mid-level, in situations where the user already has a notion at what temporal scale a pattern may be identified.

Layout

To identify low-level changes in a bottom-up based approach, several low-level electrode views need to be examined simultaneously. The system presents the views in a from-left-to-right order, synchronized with the timeline. Visual scalability is problematic as screen resolution is small relative to data resolution. Therefore, ECoG Cluster Flow utilizes a space-saving layout to achieve a tight synchronization between the spatial (electrode) view and temporal attributes (cluster evolution view) of the data. Each electrode view is ordered alternatively above and below the cluster evolution view, see Fig. 8, timesteps 3–10, to achieve the desired integration. The cluster evolution view is expanded or contracted based on the combined space allocated to all electrode views. To emphasize the granularity of the electrode view, the space assigned to each electrode view is proportional to the corresponding number of time points.

We use the formula

$$ w_{i} = w_{min}\times S_{f}, S_{f} \geq 1 $$

to define the width w _i of each electrode view, and

$$S_{f} = C \times \frac{W_{max}}{w_{min}\times max(N_{x_{1}},N_{x_{2}})} $$

to define the expansion factor, S _f , for each electrode view, The variables \(N_{x_{1}}\) and \(N_{x_{2}}\) represent the numbers of the electrode views for the x ₁− and x ₂− axes, representing the “above” and “below” cluster evolution views, respectively, The value of W _max is the maximally possible horizontal screen display width, C is a user-specified constant, and w _min defines the minimal size of each electrode view.

Synthetic dataset

In our setting, electrodes have (1) three known modes of electrical activity, (2) known intervals of activation patterns, and (3) known likelihood values for the K-cluster heatmap. The controlled data parameters allow us to investigate the features of the visualization outside the context of noisy brain recordings.

We create a dynamic network with 54 electrodes with activation values 0.9, 0.6, 0.3 activation values for 30 time steps, and we specify locations of the electrodes with our system. We keep the number of clusters constant for each timestep, and we define initial clusters for all electrodes in the 30 timesteps.

Spatio-temporal analysis:

We start our data exploration with the evolution view, looking for general evolution trends in the data. In this view, three distinct clusters (colored in red, green and blue in Fig. 9) emerge and remain stable throughout timesteps (0–9) (in Fig. 9 b). These clusters become randomly distributed at (10 to 19) to regain their stable configuration at (20 to 30).

To further examine the spatial configuration of these patterns, we employed a top-down approach exploring various temporal scales progressively to identify a consistent activation pattern (similar activation patterns in one electrode view) across all electrode views. At a granularity of ten (ten-time points in one electrode view, Fig. 9 a), persistent electrical patterns in each electrode view were found, e.g., glyphs in electrode views one and three were fully activated and deactivated, respectively. The electrode view two, on the other hand, showed a combination of activated and unactivated patterns.

To explore the intricate low-level activational and modular patterns that caused the temporal state change from unactivated to activated state, the granularity of the system was reduced to one (Fig. 9 b). When examining timesteps (9, 10, 11), an emergent focal activation point (annotated in Fig. 9 c) in the lower-right corner of the electrode view was evident. Further examination of subsequent timesteps revealed the progressive dominance of the red cluster over the region (Fig. 9 c, views 9, 10 and 11). In summary, our combined visual analysis approach helped us categorize temporal states and identify low-level changes and their dependencies with high-level state changes (Additional file 1).

Additional file 1: Demo of the tool with artificial dataset DemoVideo.mp4, A video demonstrating our tool in use with synthetic datasets. (MP4 34,764 kb)

Epileptic seizure dataset

Epilepsy is a neurological condition where the normal functioning of the brain is disrupted due to sudden bursts of electrical activity emanating from a certain region of the brain, i.e., seizure-initiating foci. This disruption is characterized by changes in the brain’s modular organization over time [50]. Exploring these differences may provide insight into the genesis and development of the seizures over time [50]. The neuroscientists on our team are primarily interested in: 1) identifying the focal site of seizure genesis and its spatial progression over various stages of the seizure and 2) identifying distinct spatio-temporal evolution patterns that characterize the onset and propagation of the seizure.

Data The raw signal data from the ECoG electrode array was statistically analyzed to provide two spatio-temporal graph datasets using different pre-processing steps: a high-gamma dataset at a frequency ranging from 70 to 170 Hz, capturing multi-unit neuronal spiking, and a full-range dataset, averaging all frequencies captured by the recording device. We derived communities independently at each timestep using the consensus clustering algorithm [42] that automatically fits the number of clusters detected. (We note that the issue of extracting the ‘correct’communities from time-varying graphs is more of a question of the statistical data analysis algorithms used in the graph formation and community detection algorithms, than the visualization there of). With these clusters as input, our system detects and visualizes the dynamic community results in the cluster evolution and spatial view.