Consensus embedding: theory, algorithms and application to segmentation and classification of biomedical data
 Satish Viswanath^{1}Email author and
 Anant Madabhushi^{1}Email author
https://doi.org/10.1186/147121051326
© Viswanath and Madabhushi; licensee BioMed Central Ltd. 2012
Received: 23 August 2011
Accepted: 8 February 2012
Published: 8 February 2012
Abstract
Background
Dimensionality reduction (DR) enables the construction of a lower dimensional space (embedding) from a higher dimensional feature space while preserving objectclass discriminability. However several popular DR approaches suffer from sensitivity to choice of parameters and/or presence of noise in the data. In this paper, we present a novel DR technique known as consensus embedding that aims to overcome these problems by generating and combining multiple lowdimensional embeddings, hence exploiting the variance among them in a manner similar to ensemble classifier schemes such as Bagging. We demonstrate theoretical properties of consensus embedding which show that it will result in a single stable embedding solution that preserves information more accurately as compared to any individual embedding (generated via DR schemes such as Principal Component Analysis, Graph Embedding, or Locally Linear Embedding). Intelligent subsampling (via meanshift) and code parallelization are utilized to provide for an efficient implementation of the scheme.
Results
Applications of consensus embedding are shown in the context of classification and clustering as applied to: (1) image partitioning of white matter and gray matter on 10 different synthetic brain MRI images corrupted with 18 different combinations of noise and bias field inhomogeneity, (2) classification of 4 highdimensional geneexpression datasets, (3) cancer detection (at a pixellevel) on 16 image slices obtained from 2 different highresolution prostate MRI datasets. In over 200 different experiments concerning classification and segmentation of biomedical data, consensus embedding was found to consistently outperform both linear and nonlinear DR methods within all applications considered.
Conclusions
We have presented a novel framework termed consensus embedding which leverages ensemble classification theory within dimensionality reduction, allowing for application to a wide range of highdimensional biomedical data classification and segmentation problems. Our generalizable framework allows for improved representation and classification in the context of both imaging and nonimaging data. The algorithm offers a promising solution to problems that currently plague DR methods, and may allow for extension to other areas of biomedical data analysis.
Background
The analysis and classification of highdimensional biomedical data has been significantly facilitated via the use of dimensionality reduction techniques, which allow classifier schemes to overcome issues such as the curse of dimensionality. This is an issue where the number of variables (features) is disproportionately large compared to the number of training instances (objects) [1]. Dimensionality reduction (DR) involves the projection of data originally represented in a Ndimensional (ND) space into a lower ndimensional (nD) space (known as an embedding) such that n << N. DR techniques are broadly categorized as linear or nonlinear, based on the type of projection method used.
Linear DR techniques make use of simple linear projections and consequently linear cost functions. An example of a linear DR scheme is Principal Component Analysis [2] (PCA) which projects data objects onto the axes of maximum variance. However, maximizing the variance within the data best preserves class discrimination only when distinct separable clusters are present within the data, as shown in [3]. In contrast, nonlinear DR involves a nonlinear mapping of the data into a reduced dimensional space. Typically these methods attempt to project data so that relative local adjacencies between high dimensional data objects, rather than some global measure such as variance, are best preserved during data reduction from N to nD space [4]. This tends to better retain classdiscriminatory information and may also account for any nonlinear structures that exist in the data (such as manifolds), as illustrated in [5]. Examples of these techniques include locally linear embedding [5] (LLE), graph embedding [6] (GE), and isometric mapping [7] (ISOMAP). Recent work has shown that in several scenarios, classification accuracy may be improved via the use of nonlinear DR schemes (rather than linear DR) for geneexpression data [4, 8] as well as medical imagery [9, 10].
However, typical DR techniques such as PCA, GE, or LLE may not guarantee an optimum result due to one or both of the following reasons:

Noise in the original ND space tends to adversely affect class discrimination, even if robust features are used (as shown in [11]). A single DR projection may also fail to account for such artifacts (demonstrated in [12, 13]).

Sensitivity to choice of parameters being specified during projection; e.g. in [14] it was shown that varying the neighborhood parameter in ISOMAP can lead to significantly different embeddings.
In this paper, we present a novel DR scheme known as consensus embedding which aims to overcome the problems of sensitivity to noise and choice of parameters that plague several popular DR schemes [12–14]. The spirit behind consensus embedding is to construct a single stable embedding by generating and combining multiple uncorrelated, independent embeddings; the hypothesis being that this single stable embedding will better preserve specific types of information in the data (such as classbased separation) as compared to any of the individual embeddings. Consensus embedding may be used in conjunction with either linear or nonlinear DR methods and, as we will show, is intended to be easily generalizable to a large number of applications and problem domains. In this work, we will demonstrate the superiority of the consensus embedding representation for a variety of classification and clustering applications.
The labels so obtained for each object (pixel) are then visualized in the image shown in Figure 1(c), where the color of the pixel corresponds to its cluster label. The 2 colors in Figure 1(c) hence correspond to the 2 classes (clusters) obtained. No discernible regions are observable in this figure. Application of DR (via GE) reduces the data to a n = 2D space, where the graph embedding algorithm [6] nonlinearly projects the data such that the object classes are maximally discriminable in the reduced dimensional space. However, as seen in Figure 1(d), clustering this reduced embedding space does not yield any obviously discernible image partitions either.
By plotting all the objects onto 2D plots using only the RG (Figure 1(e)) and RB (Figure 1(f)) color channels respectively, we can see that separation between the two classes exists only along the R axis. In contrast, the 2D GB plot (Figure 1(g)) shows no apparent separation between the classes. Combining 1D embeddings obtained via applying graph embedding to Figures 1(e) and 1(f), followed by unsupervised clustering, yields the consensus embedding result shown in Figure 1(h). Consensus embedding clearly results in superior background/foreground partitioning compared to the results shown in Figures 1(c),(d).
Related Work and Significance
Classifier and clustering ensembles
Researchers have attempted to address problems of classifier sensitivity to noise and choice of parameters via the development of classifier ensemble schemes, such as Boosting [16] and Bagging [17]. These classifier ensembles guarantee a lower error rate as compared to any of the individual members (known as "weak" classifiers), assuming that the individual weak classifiers are all uncorrelated [18]. Similarly a consensusbased algorithm has been presented [15] to find a stable unsupervised clustering of data using unstable methods such as kmeans [19]. Multiple "uncorrelated" clusterings of the data were generated and used to construct a coassociation matrix based on cluster membership of all the points in each clustering. Naturally occurring partitions in the data were then identified. This idea was further extended in [20] where a combination of clusterings based on simple linear transformations of highdimensional data was considered. Note that ensemble techniques thus (1) make use of uncorrelated, or relatively independent, analyses (such as classifications or projections) of the data, and (2) combine multiple analyses (such as classifications or projections) to enable a more stable result.
Improved DR schemes to overcome parameter sensitivity
As shown by [7], linear DR methods such as classical multidimensional scaling [21] are unable to account for nonlinear proximities and structures when calculating an embedding that best preserves pairwise distances between data objects. This led to the development of nonlinear DR methods such as LLE [5] and ISOMAP [7] which make use of local neighborhoods to better calculate such proximities. As previously mentioned, DR methods are known to suffer from certain shortcomings (sensitivity to noise and/or change in parameters). A number of techniques have recently been proposed to overcome these shortcomings.
In [22, 23] methods were proposed to choose the optimal neighborhood parameter for ISOMAP and LLE respectively. This was done by first constructing multiple embeddings based on an intelligently selected subset of parameter values, and then choosing the embedding with the minimum residual variance. Attempts have been made to overcome problems due to noisy data by selecting data objects known to be most representative of their local neighborhood (landmarks) in ISOMAP [24], or estimating neighborhoods in LLE via selection of data objects that are unlikely to be outliers (noise) [13]. Similarly, graph embedding has also been explored with respect to issues such as the scale of analysis and determining accurate groups in the data [25]. However, all of these methods require an exhaustive search of the parameter space in order to best solve the specific problem being addressed. Alternatively, one may utilize class information within the supervised variants [26, 27] of ISOMAP and LLE which attempt to construct weighted neighborhood graphs that explicitly preserve class information while embedding the data.
Learning in the context of dimensionality reduction
The application of classification theory to DR has begun to be explored recently. Athitsos et al presented a nearest neighbor retrieval method known as BoostMap [28], in which distances from different reference objects are combined via boosting. The problem of selecting and weighting the most relevant distances to reference objects was posed in terms of classification in order to utilize the Adaboost algorithm [16], and BoostMap was shown to improve the accuracy and speed of overall nearest neighbor discovery compared to traditional methods. DR has also previously been formulated in terms of maximizing the entropy [29] or via a simultaneous dimensionality reduction and regression methodology involving Bayesian mixture modeling [30]. The goal in such methods is to probabilistically estimate the relationships between points based on objective functions that are dependent on the data labels [29]. These methods have been demonstrated in the context of application of PCA to nonlinear datasets [30]. More recently, multiview learning algorithms [31] have attempted to address the problem of improving the learning ability of a system by considering several disjoint subsets of features (views) of the data. The work most closely related to our own is that of [32] in the context of web data mining via multiview learning. Given that a hidden pattern exists in a dataset, different views of this data are each embedded and transformed such that known domain information (encoded via pairwise link constraints) is preserved within a common frame of reference. The authors then solve for a consensus pattern which is considered the best approximation of the underlying hidden pattern being solved for. A similar idea was examined in [33, 34] where 1D projections of image data were coregistered in order to better perform operations such as imagebased breathing gating as well as multimodal registration. Unlike consensus embedding, these algorithms involve explicit transformations of embedding data to a target frame of reference, as well as being semisupervised in encoding specific link constraints in the data.
Intuition and significance of consensus embedding
In this paper we present a novel DR scheme (consensus embedding) that involves first generating and then combining multiple uncorrelated, independent (or base) nD embeddings. These base embeddings may be obtained via either linear or nonlinear DR techniques being applied to a large ND feature space. Note that we use the terms "uncorrelated, independent" with reference to the method of constructing base embeddings; similar to their usage in ensemble classification literature [18]. Indeed, techniques to generate multiple base embeddings may be seen to be analogous to those for constructing classifier ensembles. In the latter, base classifiers with significant variance can be generated by varying the parameter associated with the classification method (k in k NN classifiers [35]) or by varying the training data (combining decision trees via Bagging [17]). Previously, a consensus method for LLE was examined in [36] with the underlying hypothesis that varying the neighborhood parameter (κ) will effectively generate multiple uncorrelated, independent embeddings for the purposes of constructing a consensus embedding. The combination of such base embeddings for magnetic resonance spectroscopy data was found to result in a lowdimensional data representation which enabled improved discrimination of cancerous and benign spectra compared to using any single application of LLE. In this work we shall consider an approach inspired by random forests [37] (which in turn is a modification of the Bagging algorithm [17]), where variations within the feature data are used to generate multiple embeddings which are then combined via our consensus embedding scheme. Additionally, unlike most current DR approaches which require tuning of associated parameters for optimal performance in different datasets, consensus embedding offers a methodology that is not significantly sensitive to parameter choice or dataset type.

■ A novel DR approach which generates and combines embeddings.

■ A largely parameter invariant scheme for dimensionality reduction.

■ A DR scheme easily applicable to a wide variety of pattern recognition problems including image partitioning, data mining, and high dimensional data classification.
The organization of the rest of this paper is as follows. In Section 2 we will examine the theoretical grounding and properties of consensus embedding, followed by algorithms to efficiently implement the consensus embedding scheme. In Section 3 we show the application of consensus embedding in the context of (1) partitioning of synthetic as well as clinical images, and (2) classification of geneexpression studies. Quantitative and qualitative results of this evaluation, as well as discussion of the results and concluding remarks, are presented in Section 4.
Methods
Theory of Consensus Embedding
Notation and symbols
ℝ^{ N }  High(N)dimensional space  ℝ^{ n }  Low(n)dimensional space 

c, d, e  Objects in set C  Z  Number of unique triplets in C 
F(c)  Highdimensional feature vector  X(c)  Embedding vector 
Λ^{ cd }  Pairwise relationship in ℝ^{ N }  δ ^{ cd }  Pairwise relationship in ℝ^{ n } 
Δ(c, d, e)  Triangle relationship (Defn. 1)  ψ^{ ES }(ℝ^{ n })  Embedding strength (Defn. 2) 
${\widehat{\mathbb{R}}}^{n}$  True embedding (Defn. 3)  ${\widehat{\delta}}^{cd}$  Pairwise relationship in ${\widehat{\mathbb{R}}}^{n}$ 
${\ddot{\mathbb{R}}}^{n}$  Strong embedding (Defn. 4)  ${\dot{\mathbb{R}}}^{n}$  Weak embedding 
${\stackrel{\u0303}{\mathbb{R}}}^{n}$  Consensus embedding (Defn. 5)  ${\stackrel{\u0303}{\delta}}^{cd}$  Pairwise relationship in ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ 
M  Number of generated embeddings  K  Number of selected embeddings 
R  Number of objects in C  $\stackrel{\u0303}{\mathbf{X}}\left(c\right)$  Consensus embedding vector 
Preliminaries
An object shall be referred to by its label c and is defined as a point in an Ndimensional space ℝ ^{ N } . It is represented by an Ntuple F(c) comprising its unique Ndimensional coordinates. In a subspace ℝ ^{ n } ⊂ ℝ ^{ N } such that n << N, this object c in a set C is represented by an ntuple of its unique ndimensional coordinates X(c). ℝ ^{ n } is also known as the embedding of objects c ∈ C and is always calculated via some projection of ℝ ^{ N } . For example in the case of ℝ^{3}, we can define F(c) = {f_{1}, f_{2}, f_{3}} based on the coordinate locations (f_{1}, f_{2}, f_{3}) on each of the 3 axes for object c ∈ C. The corresponding embedding vector of c ∈ C in ℝ^{2} will be X(c) = {e_{1}, e_{2}} with coordinate axes locations (e_{1}, e_{2}). Note that in general, determining the target dimensionality (n) for any ℝ ^{ N } may be done by a number of algorithms such as the one used in this work [38].
The notation Λ ^{ cd } , henceforth referred to as the pairwise relationship, will represent the relationship between two objects c, d ∈ C with corresponding vectors F(c), F(d) ∈ ℝ ^{ N } . Similarly, the notation δ^{ cd } will be used to represent the pairwise relationship between two objects c, d ∈ C with embedding vectors X(c), X(d) ∈ ℝ ^{ n } . We assume that this relationship satisfies the three properties of a metric (e.g. Euclidean distance). Finally, a triplet of objects c, d, e ∈ C is referred to as an unique triplet if c ≠ d, d ≠ e, and c ≠ e. Unique triplets will be denoted simply as (c, d, e).
Definitions
Definition 1 The function Δ defined on a unique triplet (c, d, e) is called a triangle relationship, Δ(c, d, e), if when Λ ^{ cd } < Λ ^{ ce } and Λ ^{ cd } < Λ ^{ de }, then δ^{ cd } < δ^{ ce } and δ^{ cd } < δ^{ de } .
For objects c, d, e ∈ C whose relative pairwise relationships in ℝ ^{ N } are preserved in ℝ ^{ n } , the triangle relationship Δ(c, d, e) = 1. For ease of notation, the triangle relationship Δ(c, d, e) will be referred to as Δ where appropriate. Note that for a set of R unique objects (R = C, . is cardinality of a set), $Z=\frac{R!}{3!\left(R3\right)!}$ unique triplets may be formed.
Definition 2 Given Z unique triplets (c, d, e) ∈ C and an embedding ℝ ^{ n } of all objects c, d, e ∈ C, the associated embedding strength${\psi}^{ES}\left({\mathbb{R}}^{n}\right)=\frac{{\mathrm{\Sigma}}_{C}\mathrm{\Delta}\left(c,d,e\right)}{Z}$.
The embedding strength (ES) of an embedding ℝ ^{ n } , denoted ψ^{ ES } (ℝ ^{ n } ), is hence the fraction of unique triplets (c, d, e) ∈ C for which Δ(c, d, e) = 1.
Definition 3 A true embedding, ${\widehat{\mathbb{R}}}^{n}$, is an embedding for which${\psi}^{ES}\left({\widehat{\mathbb{R}}}^{n}\right)=1$.
A true embedding ${\widehat{\mathbb{R}}}^{n}$ is one for which the triangle relationship is satisfied for all unique triplets (c, d, e) ∈ C, hence perfectly preserving all pairwise relationships from ℝ ^{ N } to ${\widehat{\mathbb{R}}}^{n}$. Additionally, for all objects c, d ∈ C in ${\widehat{\mathbb{R}}}^{n}$, the pairwise relationship is denoted as ${\widehat{\delta}}^{cd}$.
Note that according to Definition 3, the most optimal true embedding may be considered to be the original ℝ ^{ N } itself, i.e. ${\widehat{\delta}}^{cd}={\mathrm{\Lambda}}^{cd}$. However, as ℝ ^{ N } may not be optimal for classification (due to the curse of dimensionality), we are attempting to approximate a true embedding as best possible in nD space. Note that multiple true embeddings in nD space may be calculated from a single ℝ ^{ N } ; any one of these may be chosen to calculate ${\widehat{\delta}}^{cd}$.
More details on this formulation may be found in the Appendix. Note that the probability in Equation 1 is binomial as the complementary probability to p(Δc, d, e, ℝ ^{ n } ) (i.e. the probability that Δ(c, d, e) ≠ 1 for any unique triplet (c, d, e) ∈ C in any ℝ ^{ n } ) is given by 1  p(Δc, d, e, ℝ ^{ n } ) (in the case of binomial probabilities, event outcomes can be broken down into two probabilities which are complementary, i.e. they sum to 1).
Definition 4 A strong embedding, ${\ddot{\mathbb{R}}}^{n}$, is an embedding for which ${\psi}^{ES}\left({\ddot{\mathbb{R}}}^{n}\right)>\theta $.
In other words, a strong embedding is defined as one which accurately preserves the triangle relationship for more than some significant fraction (θ) of the unique triplets of objects c, d, e ∈ C that exist. An embedding ℝ ^{ n } which is not a strong embedding is referred to as a weak embedding, denoted as ${\dot{\mathbb{R}}}^{n}$.
We can calculate multiple uncorrelated (i.e. independent) embeddings from a single ℝ ^{ N } which may be denoted as ${\mathbb{R}}_{m}^{n},m\in \left\{1,\dots ,M\right\}$, where M is total number of possible uncorrelated embeddings. Note that both strong and weak embeddings will be present among all of the M possible embeddings. All objects c, d ∈ C can then be characterized by corresponding embedding vectors ${\mathbf{X}}_{m}\left(c\right),{\mathbf{X}}_{m}\left(d\right)\in {\mathbb{R}}_{m}^{n}$ with corresponding pairwise relationship ${\delta}_{m}^{cd}$. Given multiple ${\delta}_{m}^{cd}$, we can form a distribution $p\left(X={\delta}_{m}^{cd}\right)$, over all M embeddings. Our hypothesis is that the maximum likelihood estimate (MLE) of $p\left(X={\delta}_{m}^{cd}\right)$, denoted as ${\stackrel{\u0303}{\delta}}^{cd}$, will approximate the true pairwise relationship ${\widehat{\delta}}^{cd}$ for objects c, d ∈ C.
Definition 5 An embedding ℝ^{ n }is called a consensus embedding, ${\stackrel{\u0303}{\mathbb{R}}}^{n}$, if for all objects $c,d\in C,{\delta}^{cd}={\stackrel{\u0303}{\delta}}^{cd}$.
We denote the consensus embedding vectors for all objects c ∈ C by $\stackrel{\u0303}{\mathbf{X}}\left(c\right)\in {\stackrel{\u0303}{\mathbb{R}}}^{n}$. Additionally, from Equation 1, $p\left(\mathrm{\Delta}c,d,e,{\stackrel{\u0303}{\mathbb{R}}}^{n}\right)$ represents the probability that Δ(c, d, e) = 1 for any (c, d, e) ∈ C in ${\stackrel{\u0303}{\mathbb{R}}}^{n}$.
Necessary and sufficient conditions for consensus embedding
While ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ is expected to approximate ${\widehat{\mathbb{R}}}^{n}$ as best possible, it cannot be guaranteed that ${\psi}^{ES}\left({\stackrel{\u0303}{\mathbb{R}}}^{n}\right)=1$ as this is dependent on how well ${\stackrel{\u0303}{\delta}}^{cd}$ approximates ${\widehat{\delta}}^{cd}$, for all objects c, d ∈ C. ${\stackrel{\u0303}{\delta}}^{cd}$ may be calculated inaccurately as a result of considering pairwise relationships derived from weak embeddings, ${\dot{\mathbb{R}}}^{n}$, present amongst the M embeddings that are generated. As Proposition 1 and Lemma 1 below demonstrate, in order to ensure that ${\psi}^{ES}\left({\stackrel{\u0303}{\mathbb{R}}}^{n}\right)\to 1$, ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ must be constructed from a combination of multiple strong embeddings ${\ddot{\mathbb{R}}}^{n}$ alone, so as to avoid including weak embeddings.
Proposition 1 If K ≤ M independent, strong embeddings${\mathbb{R}}_{k}^{n},\phantom{\rule{2.77695pt}{0ex}}k\in \left\{1,\dots ,K\right\}$, with a constant$p\left(\mathrm{\Delta}c,d,e,{\mathbb{R}}_{k}^{n}\right)$that Δ(c, d, e) = 1 for all (c, d, e) ∈ C, are used to calculate${\stackrel{\u0303}{\mathbb{R}}}^{n}$, ${\psi}^{ES}\left({\stackrel{\u0303}{\mathbb{R}}}^{n}\right)\to 1$as K → ∞.
where $\alpha =p\left(\mathrm{\Delta}c,d,e,{\mathbb{R}}_{k}^{n}\right)$ (Equation 1) is considered to be constant. Based on Equation 2, as K → ∞, $p\left(\mathrm{\Delta}c,d,e,{\stackrel{\u0303}{\mathbb{R}}}^{n}\right)\to 1$, which in turn implies that ${\psi}^{ES}\left({\stackrel{\u0303}{\mathbb{R}}}^{n}\right)\to 1$; therefore ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ approaches ${\widehat{\mathbb{R}}}^{n}$. □
Proposition 1 demonstrates that for a consensus embedding to be strong, it is sufficient that strong embeddings be used to construct it. Note that as K → M, $p\left(\mathrm{\Delta}c,d,e,{\stackrel{\u0303}{\mathbb{R}}}^{n}\right)>>p\left(\mathrm{\Delta}c,d,e,{\mathbb{R}}_{k}^{n}\right)$. In other words, if $p\left(\mathrm{\Delta}c,d,e,{\mathbb{R}}_{k}^{n}\right)>\theta ,p\left(\mathrm{\Delta}c,d,e,{\stackrel{\u0303}{\mathbb{R}}}^{n}\right)>>\theta $. Based on Equation 1 and Definitions 2, 4, this implies that as K → M, ${\psi}^{Es}\left({\stackrel{\u0303}{\mathbb{R}}}^{n}\right)>>\theta $. Lemma 1 below demonstrates the necessary nature of this condition i.e. if weak embeddings are considered when constructing ${\stackrel{\u0303}{\mathbb{R}}}^{n}$, ${\psi}^{Es}\left({\stackrel{\u0303}{\mathbb{R}}}^{n}\right)<<\theta $ (it will be a weak embedding).
Lemma 1 If K ≤ M independent, weak embeddings${\mathbb{R}}_{k}^{n},k\in \left\{1,\dots ,K\right\}$, with${\psi}^{ES}\left({\mathbb{R}}_{k}^{n}\right)\le \theta $, are used to calculate${\stackrel{\u0303}{\mathbb{R}}}^{n}$, then${\psi}^{Es}\left({\stackrel{\u0303}{\mathbb{R}}}^{n}\right)<<\theta $.
Proof. From Equation 1 and Definitions 2, 4, if ${\psi}^{ES}\left({\mathbb{R}}_{k}^{n}\right)\le \theta $, then $p\left(\mathrm{\Delta}c,d,e,{\mathbb{R}}_{k}^{n}\right)\le \theta $. Substituting $p\left(\mathrm{\Delta}c,d,e,{\mathbb{R}}_{k}^{n}\right)$ in Equation 2, will result in $p\left(\mathrm{\Delta}c,d,e,{\stackrel{\u0303}{\mathbb{R}}}^{n}\right)<<\theta $. Thus ${\psi}^{Es}\left({\stackrel{\u0303}{\mathbb{R}}}^{n}\right)<<\theta $, and ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ will be weak. □
Proposition 1 and Lemma 1 together demonstrate the necessary and sufficient nature of the conditions required to construct a consensus embedding: that if a total of M base embeddings are calculated from a single ℝ ^{ N } , some minimum number of strong embeddings (K ≤ M) must be considered to construct a ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ that is a strong embedding. Further, a ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ so constructed will have an embedding strength $\psi \left({\stackrel{\u0303}{\mathbb{R}}}^{n}\right)$ that will increase significantly as we include more strong embeddings in its computation. Appendix B demonstrates an additional property of ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ showing that it preserves information from ℝ ^{ N } with less inherent error than any ℝ ^{ n } used in its construction.
Algorithms and Implementation
Based on Proposition 1, 3 distinct steps are typically required for calculating a consensus embedding. First, we must generate a number of base embeddings (M), the steps for which are described in CreateEmbed. We then select for strong embeddings from amongst M base embeddings generated, described in SelEmbed. We will also discuss criteria for selecting strong embeddings. Finally, selected embeddings are combined to result in the final consensus embedding representation as explained in CalcConsEmbed. We also discuss some of the computational considerations of our implementation.
Creating ndimensional data embeddings
One of the requirements for consensus embedding is the calculation of multiple uncorrelated, independent embeddings ℝ ^{ n } from a single ℝ ^{ N } . This is also true of ensemble classification systems such as Boosting [16] and Bagging [17] which require multiple uncorrelated, independent classifications of the data to be generated prior to combination. As discussed previously, the terms "uncorrelated, independent" are used by us with reference to the method of constructing embeddings, as borrowed from ensemble classification literature [18]. Similar to random forests [37], we make use of a feature space perturbation technique to generate uncorrelated (base) embeddings. This is implemented by first creating M bootstrapped feature subsets of V features each (every subset η_{ m } , m ∈ {1, . . . , M} containing $\left({}_{V}^{N}\right)$ features, no DR involved). Note, that the number of samples in each Vdimensional subset is the same as in the original Ndimensional space. Each Vdimensional η_{ m } is then embedded in nD space via DR (i.e. projecting from ℝ ^{ V } to ℝ ^{ n } ). M is chosen such that each of N dimensions appears in at least one η_{ m } .
Algorithm CreateEmbed
Input: F(c) ∈ ℝ ^{ N } for all objects c ∈ C, n
Output: ${\mathbf{X}}_{m}\left(c\right)\in {\mathbb{R}}_{m}^{n},m\in \left\{1,\dots ,M\right\}$
Data Structures: Feature subsets η_{ m } , total number of subsets M, number of features in each subset V, DR method Φ
begin
0. for m = 1 to M do
1. Select V < N features from ℝ ^{ N } , forming subset η_{ m } ;
2. Calculate ${\mathbf{X}}_{m}\left(c\right)\in {\mathbb{R}}_{m}^{n}$, for all c ∈ C using η_{ m } and method Φ;
3. endfor
end
As discussed in the introduction, multiple methods exist to generate base embeddings, such as varying a parameter associated with a method (e.g. neighborhood parameter in LLE, as shown in [36]) as well as the method explored in this paper (feature space perturbation). These methods are analogous to methods in the literature for generating base classifiers in a classifier ensemble [18], such as varying k in k NN classifiers (changing associated parameter) [39], or varying the training set for decision trees (perturbing the feature space) [37].
Selection of strong embeddings
Having generated M base embeddings, we first calculate their embedding strengths ${\psi}^{ES}\left({\mathbb{R}}_{m}^{n}\right)$ for all ${\mathbb{R}}_{m}^{n},m\in \left\{1,\dots ,M\right\}$. The calculation of ψ^{ ES } can be done via performance evaluation measures such as those described below, based on the application and prior domain knowledge. Embeddings for which ${\psi}^{ES}\left({\mathbb{R}}_{m}^{n}\right)>\theta $ are then selected as strong embeddings, where θ is a prespecified threshold.
Algorithm SelEmbed
Input: ${\mathbf{X}}_{m}\left(c\right)\in {\mathbb{R}}_{m}^{n}$ for all objects c ∈ C, m ∈ {1, . . . , M}
Output: ${\mathbf{X}}_{k}\left(c\right)\in {\mathbb{R}}_{k}^{n},k\in \left\{1,\dots ,K\right\}$
Data Structures: A list Q, embedding strength function ψ^{ ES } , embedding strength threshold θ
begin
0. for m = 1 to M do
1. Calculate ${\psi}^{ES}\left({\mathbb{R}}_{m}^{n}\right)$;
2. if${\psi}^{ES}\left({\mathbb{R}}_{m}^{n}\right)>\theta $
3. Put m in Q;
4. endif
5. endfor
6. For each element k of Q, store ${\mathbf{X}}_{k}\left(c\right)\in {\mathbb{R}}_{k}^{n}$ for all objects c ∈ C;
end
Note that while θ may be considered to be a parameter which needs to be specified to construct the consensus embedding, we have found in our experiments that the results are relatively robust to variations in θ. In general, θ may be defined based on the manner of evaluating the embedding strength, as discussed in the next section.
Evaluation of embedding strength
We present two performance measures in order to evaluate embedding strength: one measure being supervised and relying on label information; the other being unsupervised and driven by the separability of distinct clusters in the reduced dimensional embedding space. In Experiment 4 we compare the two performance measures against each other to determine their relative effectiveness in constructing a strong consensus embedding.
Supervised evaluation of embedding strength
 1.
Apply classification algorithm to C (embedded in ℝ ^{ n } ) to find T clusters (unordered, labeled set of objects), denoted via ${\widehat{\mathrm{\Psi}}}_{t},t\in \left\{1,\dots ,T\right\}$.
 2.For each ${\widehat{\mathrm{\Psi}}}_{t}$
 (a)
Calculate $DTP=\phantom{\rule{2.77695pt}{0ex}}{\widehat{\mathrm{\Psi}}}_{t}\cap {S}^{+}$.
 (b)
Calculate $DTN=\phantom{\rule{2.77695pt}{0ex}}\left(C{\widehat{\mathrm{\Psi}}}_{t}\right)\cap {S}^{}$.
 (c)
Calculate classification accuracy for ${\widehat{\mathrm{\Psi}}}_{t}$, as ${\varphi}^{Acc}\left({\widehat{\mathrm{\Psi}}}_{t}\right)=\frac{DTP+DTN}{{s}^{+}\cup {s}^{}}$.
 (a)
 3.
Calculate classification accuracy of ℝ ^{ n } as ${\varphi}^{Acc}\left({\mathbb{R}}^{n}\right)=\underset{T}{\mathrm{max}}\left[{\varphi}^{Acc}\left({\widehat{\mathrm{\Psi}}}_{t}\right)\right]$.
As classification has been done without considering label information, we must evaluate which of the clusters so obtained shows the greatest overlap with S^{+} (the class of interest). We therefore consider the classification accuracy of the cluster showing the most overlap with S^{+} as an approximation of the embedding strength of ℝ ^{ n } , i.e. ψ^{ ES } (ℝ ^{ n } ) ≈ ϕ^{ Acc } (ℝ ^{ n } ).
Unsupervised evaluation of embedding strength
 1.
Apply classification algorithm to C (embedded in ℝ ^{ n } ) to find T clusters (unordered, labeled set of objects), denoted via ${\widehat{\mathrm{\Psi}}}_{t},t\in \left\{1,\dots ,T\right\}$.
 2.
Calculate $SST={\sum}_{j=1}^{n}\left[{\sum}_{i=1}^{R}{\left(\mathbf{X}\left({c}_{i}\right)\mathbf{X}\left({c}_{j}\right)\right)}^{2}\right]$ (where X(c_{ j } ) is the mean of data values in the j^{ th } dimension).
 3.
Calculate $SSB={\sum}_{\begin{array}{c}j=1\cdot \cdot \cdot n\\ t=1\cdot \cdot \cdot T\end{array}}\left[{\sum}_{i=1}^{\left{\widehat{\mathrm{\Psi}}}_{t}\right}{\left(\mathbf{X}\left({c}_{i}\right)\mathbf{X}\left({c}_{j}\right)\right)}^{2}\right]$.
 4.
Calculate Rsquared index of ℝ ^{ n } as ${\varphi}^{RS}\left({\mathbb{R}}^{n}\right)=\frac{SSTSSB}{SST}$.
RSI may be considered both a measure of the degree of difference between clusters found in a dataset as well as measurement of the degree of homogeneity between them. The value of ϕ^{ RS } ranges between 0 and 1, where if ϕ^{ RS } = 0, no difference exists among clusters. Conversely, a value close to ϕ^{ RS } = 1 suggests welldefined, separable clusters in the embedding space. Note that when using RSI to evaluate embedding strength, it will be difficult to ensure that all selected embeddings are strong without utilizing a priori information. In such a case we can attempt to ensure that a significant majority of the embeddings selected are strong, which will also ensure that the consensus embedding ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ is strong (based off Proposition 1).
Constructing the consensus embedding
Given K selected embeddings ${\mathbb{R}}_{k}^{n},k\in \left\{1,\dots ,K\right\}$, we quantify pairwise relationships between all the objects in each ${\mathbb{R}}_{k}^{n}$ via Euclidean pairwise distances. Euclidean distances were chosen for our implementation as they are well understood, satisfy the metric assumption of the pairwise relationship, as well as being directly usable within the other methods used in this work. Ω _{ k } denotes the ML estimator used for calculating ${\stackrel{\u0303}{\delta}}^{cd}$ from K observations ${\delta}_{k}^{cd}$ for all objects c, d ∈ C.
Algorithm CalcConsEmbed
Input: ${\mathbf{X}}_{k}\left(c\right)\in {\mathbb{R}}_{k}^{n}$ for all objects c ∈ C, k ∈ {1, . . . , K}
Output: $\stackrel{\u0303}{\mathbf{X}}\left(c\right)\in {\stackrel{\u0303}{\mathbb{R}}}^{n}$
Data Structures: Confusion matrix W, ML estimator Ω, projection method γ
begin
0. for k = 1 to K do
1. Calculate W_{ k } (i, j) = X _{ k }(c)  X _{ k }(d)_{2} for all objects c, d ∈ C with indices i, j;
2. endfor
3. Apply normalization to all W_{ k } , k ∈ {1, . . . , K};
4. Obtain $\stackrel{\u0303}{W}\left(i,j\right)={\mathrm{\Omega}}_{k}\left[{W}_{k}\left(i,j\right)\right]\forall c,d\in C$;
5. Apply projection method γ to $\stackrel{\u0303}{W}$ to obtain final consensus embedding ${\stackrel{\u0303}{\mathbb{R}}}^{n}$;
end
Corresponding entries across all W_{ k } (after any necessary normalization) are used to estimate ${\stackrel{\u0303}{\delta}}^{cd}$ (and stored in $\stackrel{\u0303}{W}$). In our implementation, we have used the median as the ML estimator as (1) the median is less corruptible to outliers, and (2) the median and the expectation are interchangeable if one assumes a normal distribution [41]. In Section 3 we compare classification results using both the mean and median individually as the ML estimator. We apply a projection method γ, such as multidimensional scaling (MDS) [21], to the resulting $\stackrel{\u0303}{W}$ to embed the objects in ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ while preserving the pairwise distances between all objects c ∈ C. The underlying intuition for this final step is based on a similar approach adopted in [15] where MDS was applied to the coassociation matrix (obtained by accumulating multiple weak clusterings of the data) in order to visualize the clustering results. As $\stackrel{\u0303}{W}$ is analogous to the coassociation matrix, the projection method γ will allow us to construct the consensus embedding space ${\stackrel{\u0303}{\mathbb{R}}}^{n}$.
One can hypothesize that $\stackrel{\u0303}{W}$ is an approximation of distances calculated in the original feature space. Distances in the original feature space can be denoted as $\u0174\left(i,j\right)=\phantom{\rule{2.77695pt}{0ex}}\left\right\mathbf{F}\left(c\right)\mathbf{F}\left(d\right){}_{2}\forall c,d\in C$ with indices i, j. An alternative approach could therefore be to calculate $\u0174$ in the original feature space and apply γ to it instead. However, noise artifacts in the original feature space may prevent it from being truly optimal for analysis [11]. As we will demonstrate in Section 3, simple DR, as well as consensus DR, provide superior representations of the data (by accounting for noise artifacts) as compared to using the original feature space directly.
Computational efficiency of Consensus Embedding
The most computationally expensive operations in consensus embedding are (1) calculation of multiple uncorrelated embeddings (solved as an eigenvalue problem in O(n^{3}) time for n objects), and (2) computation of pairwise distances between all the objects in each strong embedding space (computed in time O(n^{2}) for n objects). A slight reduction in both time and memory complexity can be achieved based on the fact that distance matrices will be symmetric (hence only the upper triangular need be calculated). Additionally, multiple embeddings and distance matrices can be computed via code parallelization. However these operations still scale polynomially based on the number of objects n.
To further reduce the computational burden we embed the consensus embedding paradigm within an intelligent subsampling framework. We make use of a fast implementation [42] of the popular mean shift algorithm [43] (MS) to iteratively represent data objects via their most representative cluster centers. As a result, the space retains its original dimensionality, but now comprises only some fractional number (n/t) of the original objects. These n/t objects are used in the calculations of consensus embedding as well as for any additional analysis. A mapping (Map) is retained from all n original objects to the final n/t representative objects. We can therefore map back results and analyses from the lowest resolution (n/t objects) to the highest resolution (n objects) easily. The fewer number of objects (n/t << n) ensures that consensus embedding is computationally feasible. In our implementation, t was determined automatically based on the number of stable cluster centers detected by MS.
Algorithm ConsEmbedMS
Input: F(c) ∈ ℝ ^{ N } for all objects c ∈ C, n
Output: $\stackrel{\u0303}{\mathbf{X}}\left(c\right)\in {\stackrel{\u0303}{\mathbb{R}}}^{n}$
Data Structures: Reduced set of objects $\stackrel{\u0304}{c}\in \stackrel{\u0304}{C}$
begin
0. Apply MS [42] to ℝ ^{ N } resulting in ${\stackrel{\u0304}{\mathbb{R}}}^{N}$ for subsampled set of objects $\stackrel{\u0304}{c}\in \stackrel{\u0304}{C}$;
1. Save Map from subsampled set of objects $\stackrel{\u0304}{c}\in \stackrel{\u0304}{C}$ to original set of objects c ∈ C;
2. ${\mathbf{X}}_{m}\left(\stackrel{\u0304}{c}\right)=CreateEmbed\left(\mathbf{F}\left(\stackrel{\u0304}{c}\right){\eta}_{m},\phantom{\rule{2.77695pt}{0ex}}\Phi ,\phantom{\rule{2.77695pt}{0ex}}M,\phantom{\rule{2.77695pt}{0ex}}V\right),\forall m\in \left\{1,\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}M\right\}$;
3. ${\mathbf{X}}_{k}\left(\stackrel{\u0304}{c}\right)=SelEmbed\left({\mathbf{X}}_{m}\left(\stackrel{\u0304}{c}\right)Q,\phantom{\rule{2.77695pt}{0ex}}\psi ,\phantom{\rule{2.77695pt}{0ex}}\theta \right),\forall k\in \left\{1,\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}K\right\},\forall m\in \left\{1,\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}M\right\}$;
4. $\stackrel{\u0303}{\mathbf{X}}\left(\stackrel{\u0304}{c}\right)=CalcConsEmbed\left({\mathbf{X}}_{k}\left(\stackrel{\u0304}{c}\right)W,\phantom{\rule{2.77695pt}{0ex}}\mathrm{\Omega},\phantom{\rule{2.77695pt}{0ex}}\gamma \right),\forall k\in \left\{1,\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}K\right\}$;
5. Use MS and Map to calculate $\stackrel{\u0303}{\mathbf{X}}\left(c\right)\in {\stackrel{\u0303}{\mathbb{R}}}^{n}$ from $\stackrel{\u0303}{\mathbf{X}}\left(\stackrel{\u0304}{c}\right)\in {\stackrel{\u0304}{\mathbb{R}}}^{n}$ for all objects c ∈ C;
end
For an MRI image comprising 5589 pixels (objects) for analysis, the individual algorithms CreateEmbed, SelEmbed and CalcConsEmbed took 121.33, 12.22, and 35.75 seconds respectively to complete (on average). By implementing our meanshift optimization it took only 119 seconds (on average) for ConsEmbedMS to complete analysis of an MRI image comprising between 15,000 and 40,000 pixels (objects); a calculation that would have been computationally intractable otherwise. All experiments were conducted using MATLAB 7.10 (Mathworks, Inc.) on a 72 GB RAM, 2 quad core 2.33 GHz 64bit Intel Core 2 processor machine.
Experimental Design for Evaluating Consensus Embedding
Dataset description
Datasets
Datasets  Description  Features 

Synthetic brain MRI images  10 slices (109 × 131 comprising 5589 pixels), 6 noise levels (0%, 1%, 3%, 5%, 7%, 9%) 3 RF inhomogeneity levels (0%, 20%, 40%)  Haralick (14) 
Prostate MRI images  16 slices, 2 datasets (256 × 256 comprising 15,00040,000 pixels)  Haralick, 1st order statistical (38) 
GeneExpression data:  
Prostate Tumor  102 training, 34 testing, 12,600 genes  
Breast Cancer Relapse  78 training, 19 testing, 24,481 genes  300 most class 
Lymphoma  38 training, 34 testing, 7130 genes  informative genes 
Lung Cancer  32 training, 149 testing, 12,533 genes 
In the case of image data (brain, prostate MRI), we have derived texture features [44] on a perpixel basis from each image. These features are based on calculating statistics from a gray level intensity cooccurrence matrix constructed from the image, and were chosen due to previously demonstrated discriminability between cancerous and noncancerous regions in the prostate [45] and different types of brain matter [46] for MRI data. Following feature extraction, each pixel c in the MR image is associated with a N dimensional feature vector F(c) = [f_{ u } (c)u ∈ {1, . . . , N}] ∈ ℝ ^{ N } , where f_{ u } (c) is the response to a feature operator for pixel c. In the case of geneexpression data, every sample c is considered to be associated with a highdimensional geneexpression vector, also denoted F(c) ∈ ℝ ^{ N } .
DR methods utilized to reduce ℝ ^{ N } to ℝ ^{ n } were graph embedding (GE) [6] and PCA [2]. These methods were chosen in order to demonstrate instantiations of consensus embedding using representative linear and nonlinear DR schemes. Additionally, these methods have been leveraged both for segmentation as well as classification of similar biomedical image and bioinformatics datasets in previous work [47, 48]. The dimensionality of the embedding space, n, is calculated as the intrinsic dimensionality of ℝ ^{ N } via the method of [38]. To remain consistent with notation defined previously, the result of DR on F(c) ∈ ℝ ^{ N } is denoted X_{Φ}(c) ∈ ℝ ^{ n } , while the result of consensus DR will be denoted ${\stackrel{\u0303}{\mathbf{X}}}_{\mathrm{\Phi}}\left(c\right)\in {\stackrel{\u0303}{\mathbb{R}}}^{n}$. The subscript Φ corresponds to the DR method used, Φ ∈ {GE, PCA}. For ease of description, the corresponding classification results are denoted Ψ(F), Ψ(X_{Φ}), $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{\mathrm{\Phi}}\right)$, respectively.
Experiment 1: Synthetic MNI brain data
 (i)
the highdimensional feature space F(c) ∈ ℝ ^{ N } , N = 14,
 (ii)
simple GE on F(c), denoted X _{ GE }(c) ∈ ℝ ^{ n } , n = 3,
 (iii)
multidimensional scaling (MDS) on distances calculated directly in ℝ ^{ N } , denoted as X _{ MDS }(c) ∈ ℝ ^{ n } , n = 3 (alternative to consensus embedding, explained in Section 2),
 (iv)
consensus embedding, denoted ${\stackrel{\u0303}{\mathbf{X}}}_{GE}\left(c\right)\in {\stackrel{\u0303}{\mathbb{R}}}^{n}$, n = 3.
The final slice classification results obtained for each of these spaces are denoted as Ψ(F), Ψ(X _{ GE }), Ψ(X _{ MDS }), $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}\right)$, respectively.
Experiment 2: Comparison of ML estimators in consensus embedding
For the synthetic brain data [49], over all 18 combinations of noise and inhomogeneity and over all 10 images, we compare the use of mean and median as ML estimators in CalcConsEmbed. This is done by preserving outputs from SelEmbed and only changing the ML estimator in the CalcConsEmbed. We then compare classification accuracies for detection of white matter in each of the resulting consensus embedding representations, ${\stackrel{\u0303}{\mathbf{X}}}_{GE}^{Med}$ and ${\stackrel{\u0303}{\mathbf{X}}}_{GE}^{Mean}$ (superscript denotes choice of ML estimator).
Experiment 3: Clinical prostate MRI data
Two different prostates were imaged ex vivo using a 4 Tesla MRI scanner following surgical resection. The excised glands were then sectioned into 2D histological slices which were digitized using a whole slide scanner. Regions of cancer were determined via Haemotoxylin and Eosin (H&E) staining of the histology sections. The cancer areas were then mapped onto corresponding MRI sections via a deformable registration scheme [50]. Additional details of data acquisition are described in [45].
 (a)
Training (S^{ tr } ) and testing (S^{ te } ) sets of the data (MRI images) were created.
 (b)
For each element (image) of S^{ tr } , strong embeddings were identified based on $\theta =0.15\times \underset{M}{\mathrm{max}}\left[\psi \left({\mathbb{R}}_{m}^{n}\right)\right]$.
 (c)
Those embeddings voted as being strong across all the elements (images) in S^{ tr } were then identified and selected.
 (d)
For the data (images) in S^{ te } , corresponding embeddings were then combined (via CalcConsEmbed) to yield the final consensus embedding result.
A leaveoneout crossvalidation strategy was employed in this experiment. A comparison is made between the pixellevel classifications for (1) simple GE denoted as Ψ(X _{ GE }), and (2) consensus GE denoted as $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}\right)$.
Experiment 4: Geneexpression data
Four publicly available binary class geneexpression datasets were obtained^{2} with corresponding class labels for each sample [4]; the purpose of the experiment being to differentiate the two classes in each dataset. This data comprises the geneexpression vectorial data profiles of normal and cancerous samples for each disease listed in Table 2, where the total number of samples range from 72 to 181 patients and the number of corresponding features range from 7130 to 24,481 genes or peptides. All 4 data sets comprise independent training (S^{ tr } ) and testing (S^{ te } ) subsets, and these were utilized within a supervised framework for constructing and evaluating the consensus embedding representation.
 (a)
First, CreateEmbed is run concurrently on data in S^{ tr } and S^{ te } , such that the same subsets of features are utilized when generating base embeddings for each of S^{ tr } and S^{ te } .
 (b)
SelEmbed is then executed on base embeddings generated from S^{ tr } alone, thus selecting strong embeddings from amongst those generated. Strong embeddings were defined based on $\theta =0.15\times \underset{M}{\mathrm{max}}\left[\psi \left({\mathbb{R}}_{m}^{n}\right)\right]$.
 (c)
Corresponding (selected) embeddings for data in S^{ te } are then combined within CalcConsEmbed to obtain the final consensus embedding vectors denoted as ${\stackrel{\u0303}{\mathbf{X}}}_{\mathrm{\Phi}}\left(c\right)\in {\stackrel{\u0303}{\mathbb{R}}}^{n}$, Φ ∈ {GE, PCA}, n = 4.
For this dataset, both supervised (via clustering classification accuracy, superscript S) and unsupervised (via RSI, superscript US) measures of embedding strength were evaluated in terms of the classification accuracy of the corresponding consensus embedding representations.
In lieu of comparative DR strategies, a semisupervised variant of GE [52] (termed SSAGE) was implemented, which utilizes label information when constructing the embedding. Within this scheme, higher weights are given to withinclass points and lower weights to points from different classes. When running SSAGE, both S^{ tr } and S^{ te } were combined into a single cohort of data, and labels corresponding to S^{ tr } alone were revealed to the SSAGE algorithm.
An additional comparison was conducted against a supervised random forestbased k NN classifier operating in the original feature space to determine whether DR provided any advantages in the context of highdimensional biomedical data. This was implemented by training a k NN classifier on each of the feature subsets for S^{ tr } (that were utilized in Create Embed), but without performing DR on the data. Each such k NN classifier was then used to classify corresponding data in S^{ te } . The final classification result for each sample in S^{ te } is based on ensemble averaging to calculate the probability of a sample belonging to the target class. Classifications compared in this experiment were Ψ(F), Ψ(X _{ SSGE }), $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{S}\right)$, $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{PCA}^{S}\right)$, $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{US}\right)$, $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{PCA}^{US}\right)$, respectively.
Classification
For image data (brain, prostate MRI), classification was done via replicated kmeans clustering [15], while for geneexpression data, classification was done via hierarchical clustering [53]. The choice of clustering algorithm was made based on the type of data being considered in each of the different experiments, as well as previous work in the field. Note that both these clustering techniques do not consider class label information while classifying the data, and have been demonstrated as being deterministic in nature (hence ensuring reproducible results). The motivation in using such techniques for classification was to ensure that no classifier bias or fitting optimization was introduced during evaluation. As our experimental intent was purely to examine improvements in class separation offered by the different data representations, all improvements in corresponding classification accuracies may be directly attributed to improved class discriminability in the corresponding space being evaluated (without being dependent on optimizing the technique used for classification).
Evaluating and visualizing results
To visualize classification results as region partitions on the images (brain, prostate MRI), all the pixels were plotted back onto the image and assigned colors based on their classification label membership. Similar to the partitioning results shown in Figure 1, pixels of the same color were considered to form specific regions. For example, in Figure 1(h), pixels colored green were considered to form the foreground region, while pixels colored red were considered to form the background.
Classification accuracy of clustering results for images as well as geneexpression data can be quantitatively evaluated as described previously (Section 2). Image region partitioning results as well as corresponding classification accuracies of the different methods (GE, PCA, consensus embedding) were used to determine what improvements are offered by consensus embedding.
Results and Discussion
Experiment 1: Synthetic MNI Brain data
WM detection results for synthetic BrainWeb data
Noise  Inhomogeneity  ϕ^{ Acc }(F)  ϕ^{ Acc }(X_{ MDS })  ϕ^{ Acc }(X_{ GE })  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}\right)$ 

0%  65.55 ± 1.84  65.55 ± 1.84  65.55 ± 1.84  66.86 ± 2.89  
0%  20%  55.75 ± 1.65  55.75 ± 1.65  55.75 ± 1.65  61.65 ± 4.58 
40%  70.03 ± 2.79  70.08 ± 2.82  51.84 ± 0.99  64.28 ± 5.93  
0%  59.78 ± 1.31  59.74 ± 1.29  74.71 ± 9.06  80.62 ± 1.03  
1%  20%  59.36 ± 1.30  59.32 ± 1.33  60.95 ± 8.67  73.07 ± 8.97 
40%  59.20 ± 1.12  59.12 ± 1.15  56.38 ± 1.53  66.46 ± 9.80  
0%  53.35 ± 1.31  53.39 ± 1.27  59.94 ± 7.00  85.38 ± 0.75  
3%  20%  55.01 ± 2.92  54.91 ± 3.11  63.88 ± 10.85  84.61 ± 0.81 
40%  57.63 ± 1.78  57.71 ± 1.67  57.33 ± 1.38  79.19 ± 7.56  
0%  62.90 ± 0.72  62.84 ± 0.66  66.67 ± 10.22  89.68 ± 1.36  
5%  20%  61.49 ± 1.38  61.49 ± 1.42  82.61 ± 7.39  86.81 ± 1.38 
40%  61.02 ± 0.99  61.03 ± 1.09  74.91 ± 9.09  81.67 ± 1.51  
0%  64.28 ± 0.71  64.26 ± 0.76  66.95 ± 6.25  87.81 ± 0.73  
7%  20%  64.07 ± 1.03  64.01 ± 0.96  74.22 ± 10.59  86.07 ± 1.05 
40%  64.05 ± 1.19  64.04 ± 1.14  64.44 ± 1.25  81.53 ± 1.57  
0%  64.96 ± 0.90  64.94 ± 0.88  66.36 ± 1.66  75.51 ± 14.35  
9%  20%  64.85 ± 0.97  64.79 ± 0.95  65.68 ± 1.32  78.18 ± 9.86 
40%  64.65 ± 0.83  64.63 ± 0.84  65.30 ± 0.74  77.83 ± 5.00 
For each combination of noise and inhomogeneity, a paired Students' ttest was conducted between ${{\varphi}^{A}}^{cc}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}\right)$ and each of ϕ^{ Acc } (F), ϕ^{ Acc } (X _{ MDS }), and ϕ^{ Acc } (X _{ GE }), with the null hypothesis being that there was no improvement via $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}\right)$ over all 10 brain images considered. $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}\right)$ was found to perform significantly better (p < 0.05) than all of Ψ(F), Ψ(X _{ MDS }), and Ψ(X _{ GE }) in 16 out of 18 combinations of noise and inhomogeneity.
Comparing ϕ^{ Acc } (F), ϕ^{ Acc } (X _{ MDS }), and ϕ^{ Acc } (X _{ GE }), it can be observed that Ψ(F) and Ψ(X _{ MDS }) perform similarly for all combinations of noise and inhomogeneity (note that the corresponding red and blue trendlines completely overlap in Figure 3). In contrast, Ψ(X _{ GE }) shows improved performance at every combination of noise and inhomogeneity as compared to either of Ψ(F) and Ψ(X _{ MDS }). $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}\right)$ was seen to significantly improve over all of Ψ(F), Ψ(X _{ MDS }), and Ψ(X _{ GE }), reflecting the advantages of consensus embedding.
Experiment 2: Comparison of ML estimators
Comparing the mean and median as ML estimators within CalcConsEmbed
Noise  Inhomogeneity  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{Med}\right)$  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{Mean}\right)$ 

0%  66.86 ± 2.89  66.89 ± 2.91  
0%  20%  61.65 ± 4.58  65.34 ± 4.12 
40%  64.28 ± 5.93  63.39 ± 6.51  
0%  80.62 ± 1.03  80.45 ± 1.07  
1%  20%  73.07 ± 8.97  77.81 ± 0.96 
40%  66.46 ± 9.80  70.56 ± 7.15  
0%  85.38 ± 0.75  85.53 ± 0.84  
3%  20%  84.61 ± 0.81  84.49 ± 0.76 
40%  79.19 ± 7.56  81.37 ± 1.39  
0%  89.68 ± 1.36  90.85 ± 1.32  
5%  20%  86.81 ± 1.38  87.01 ± 1.83 
40%  81.67 ± 1.51  81.82 ± 1.32  
0%  87.81 ± 0.73  86.17 ± 6.11  
7%  20%  86.07 ± 1.05  82.73 ± 8.23 
40%  81.53 ± 1.57  81.72 ± 1.47  
0%  75.51 ± 14.35  74.32 ± 16.11  
9%  20%  78.18 ± 9.86  73.63 ± 12.75 
40%  78.18 ± 9.86  73.63 ± 12.75 
Experiment 3: Clinical Prostate MRI data
Experiment 4: Classification of GeneExpression Data
Classification accuracies for different representation strategies for geneexpression data
Dataset  ϕ^{ Acc }(F)  ϕ^{ Acc }(X_{ SSGE })  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{PCA}^{S}\right)$  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{PCA}^{US}\right)$  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{S}\right)$  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{US}\right)$ 

Prostate Tumor  73.53  73.53  97.06  100  100  76.47 
Breast Cancer Relapse  68.42  63.16  63.16  57.89  63.16  57.89 
Lung Cancer  89.93  10.07  99.33  96.64  98.66  100 
Lymphoma  58.82  61.76  97.06  76.47  97.06  67.65 
Further, comparing the performance of supervised $\left(\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{PCA}^{S}\right),\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{S}\right)\right)$ and unsupervised $\left(\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{PCA}^{US}\right),\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{US}\right)\right)$ variants of consensus embedding demonstrates comparable performance between them, though a supervised measure of embedding strength shows a trend towards being more consistent. The relatively high performance of $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{PCA}^{US}\right)$ and $\mathrm{\Psi}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{US}\right)$ demonstrate the feasibility of a completely unsupervised framework for consensus embedding.
Variation in classification accuracy as a function of parameters for consensusPCA on geneexpression data
Dataset  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{PCA}^{S}\right)$  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{PCA}^{US}\right)$  

M = 200  M = 500  M = 1000  M = 200  M = 500  M = 1000  
Prostate Tumor  97.06  97.06  97.06  100  100  100 
Breast Cancer Relapse  57.89  63.16  57.89  57.89  57.89  52.63 
Lung Cancer  99.33  99.33  99.33  96.64  95.97  96.64 
Lymphoma  94.12  97.06  97.06  76.47  67.65  61.76 
Variation in classification accuracy as a function of parameters for consensusGE on geneexpression data
Dataset  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{S}\right)$  ${\varphi}^{Acc}\left({\stackrel{\u0303}{\mathbf{X}}}_{GE}^{US}\right)$  

M = 200  M = 500  M = 1000  M = 200  M = 500  M = 1000  
Prostate Tumor  100  100  97.06  76.47  76.47  76.47 
Breast Cancer Relapse  57.89  57.89  57.89  57.89  57.89  57.89 
Lung Cancer  98.66  98.66  97.99  100  100  90.60 
Lymphoma  61.76  97.06  55.88  67.65  67.65  67.65 
Conclusions
We have presented a novel dimensionality reduction scheme called consensus embedding which can be used in conjunction with a variety of DR methods for a wide range of highdimensional biomedical data classification and segmentation problems. Consensus embedding exploits the variance within multiple base embeddings and combines them to produce a single stable solution that is superior to any of the individual embeddings, from a classification perspective. Specifically, consensus embedding is able to preserve pairwise objectclass relationships from the high to the lowdimensional space more accurately compared to any single embedding technique. Using an intelligent subsampling approach (via meanshift) and code parallelization, computational feasibility and practicability of our method is ensured.
Results of quantitative and qualitative evaluation in over 200 experiments on toy, synthetic, and clinical images in terms of detection and classification accuracy demonstrated that consensus embedding shows significant improvements compared to traditional DR methods such as PCA. We also compared consensus embedding to using the feature space directly, as well as to using an embedding based on distance preservation directly from the feature space (via MDS [21]), and found significant performance improvements when using consensus embedding. Even though the features and classifier used in these experiments were not optimized for image segmentation purposes, consensus embedding outperforms stateoftheart segmentation schemes (graph embedding, also known as normalized cuts [6]), differences being statistically significant in all cases. Incorporating spatial constraints via algorithms such as Markov Random Fields [54] could be used to further bolster the image segmentation results via consensus embedding.
In experiments for highdimensional biomedical data analysis using geneexpression signatures, consensus embedding also demonstrated improved results compared to semisupervised DR methods (SSAGE [52]). Evaluating these results further illustrates properties of consensus embedding: (1) the consensus of multiple projections improves upon any single projection (via either linear PCA or nonlinear GE), (2) the error rate for consensus embedding is not significantly affected by parameters associated with the method, as compared to traditional DR. Finally, the lower performance of a supervised classifier using the original noisy feature space as compared to using the consensus embedding representation demonstrates the utility of DR to obtain improved representations of the data for classification.
It is however worth noting that in certain scenarios, consensus embedding may not yield optimal results. For instance, if very few embeddings are selected for consensus, the improvement in performance via consensus embedding over simple DR techniques may not be as significant. This translates to having a sparsely populated distribution for the estimation of the consensus pairwise relationship, resulting in a lower confidence being associated with it. Such a scenario may arise due to incorrectly specified selection criteria for embeddings; however, it is relatively simple to implement selftuning for the selection parameter (θ). Note we have reported results for a fixed value of θ in all our experiments/applications, further demonstrating the robustness of our methodology to choice of parameters.
In this work, consensus embedding has primarily been presented within a supervised framework (using class label information to evaluate embedding strength). Preliminary results in developing an unsupervised evaluation measure using the Rsquared cluster validity index [40] are extremely promising. However, additional tuning and testing of the measure is required to ensure robustness.
Another area of future work is developing algorithms for the generating uncorrelated, independent embeddings. This is of great importance as generating truly uncorrelated, independent embeddings will allow us to capture the information from the data better, hence ensuring in an improved consensus embedding result. As mentioned previously, methods to achieve this could include varying the parameter associated with the DR method (e.g. neighborhood parameter in LLE [5]) as well as the feature space perturbation method explored in this paper. These approaches are analogous to methods of generating weak classifiers within a classifier ensemble [18], such as varying the k parameter in k NN classifiers [39] or varying the training set for decision trees [37]. Note our feature space perturbation method to generate multiple, uncorrelated independent embeddings is closely related to the method used in random forests [37] to generate multiple weak, uncorrelated classifiers. Thus the embeddings we generate, as with the multiple classifiers generated in ensemble classifier schemes, are not intended to be independent in terms of information content, but rather in their method of construction.
The overarching goal of consensus embedding is to optimally preserve pairwise relationships when projecting from high to lowdimensional space. In this work, pairwise relationships were quantified by us using the popular Euclidean distance metric. This was chosen as it is well understood in the context of these methods used within our algorithm (e.g. the use of MDS). Alternative pairwise relationship measures could include the geodesic distance [7] or the symmetrized KullbackLeibler divergence [55]. It is important to note that such measures will need to satisfy the properties of a metric to ensure that they correctly quantify both triangle as well as pairwise relationships. We currently use MDS [21] to calculate the final consensus embedding (based on the consensus pairwise distance matrix). We have chosen to use MDS due to ease of computational complexity, but this method could be replaced by a nonlinear variant instead. Finally, our intelligent subsampling approach to ensure computational feasibility comes with a caveat of the outofsample extension problem [56]. We currently handle this using a mapping of results from high to lowresolutions, but are currently identifying more sophisticated solutions. We intend to study these areas in greater detail to further validate the generalizability of consensus embedding.
Appendix A: Correspondence between Equation 1 and Definition 2
Independent of the above, we intuitively arrived at a mathematical formulation for embedding strength.
The strength of any embedding ℝ ^{ n } will depend on how well pairwise relationships are preserved from ℝ ^{ N } .
Appendix B: Properties of consensus embedding
The following proposition will demonstrate that ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ will have a lower inherent error in its pairwise relationships compared to the strong embeddings ${\mathbb{R}}_{k}^{n},k\in \left\{1,\dots ,K\right\}$, used in its construction. Note that relevant notation and definitions have been carried over from Section 2.
where E _{ cd } is the expectation of the squared error in the pairwise relationships in ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ calculated over over all pairs of objects c, d ∈ C. It is clear that if for all c, d ∈ C that ${\stackrel{\u0303}{\delta}}^{cd}={\widehat{\delta}}^{cd}$, then ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ is also a true embedding.
Proposition 2 Given K independent, strong embeddings, ${\mathbb{R}}_{k}^{n},k\in \left\{1,\dots ,K\right\}$, which are used to construct${\stackrel{\u0303}{\mathbb{R}}}^{n}$, ${\epsilon}_{K,X}\ge {\epsilon}_{\stackrel{\u0303}{X}}$.
Proposition 2 implies that ${\stackrel{\u0303}{\mathbb{R}}}^{n}$ will never have a higher error than the maximum error associated with any individual strong embedding ${\mathbb{R}}_{k}^{n},k\in \left\{1,\dots ,K\right\}$, involved in its construction. However if ε_{K,X}is low, ${\epsilon}_{\stackrel{\u0303}{X}}$ may not significantly improve on it. Similar to Bagging [17] where correlated errors across weak classifiers are preserved in the ensemble result, if the pairwise relationship ${\delta}_{k}^{cd}$ is incorrect across all K embeddings, ${\stackrel{\u0303}{\delta}}^{cd}$ will be incorrect as well. However Proposition 2 guarantees that ${\epsilon}_{\stackrel{\u0303}{X}}$ will never be worse than ε_{K,X}.
Appendix C: Practical implementation of embedding strength
While embedding strength may be seen as a generalized concept for evaluating embeddings, in this work we have examined applications of DR and consensus embedding to classifying biomedical data (Section 3). We now derive a direct relationship between embedding strength and classification accuracy, presented in Theorem 1 below.
For the purposes of the following discussion, all objects c, d, e ∈ C are considered to be associated with class labels l(c), l(d), l(e) ∈ {ω_{1}, ω_{2}}, respectively, such that if l(c) = l(d) = ω_{1} and l(e) = ω_{2} then Λ ^{ cd } < Λ ^{ ce } and Λ ^{ cd } < Λ ^{ de } . Note that ω_{1}, ω_{2} are binary class labels that can be assigned to all objects c ∈ C.
Definition 6 An unique triplet (c, d, e) ∈ C with l(c), l(d), l(e) ∈ {ω_{1}, ω_{2}} is called a class triplet, (c, d, e) _{ l }, if either l(c) ≠ l(d), or l(d) ≠ l(e), or l(c) ≠ l(e).
Thus, in a class triplet of objects, two out of three objects have the same class label but the third has a different class label, e.g. if l(c) = l(d) = ω_{1} and l(e) = ω_{2}. Further, in the specific case that Δ(c, d, e) = 1 for a class triplet (c, d, e)_{ l }, it will be denoted as Δ ^{ l } (c, d, e). For the above example of a class triplet, we know that Λ ^{ cd } < Λ ^{ ce } and Λ ^{ cd } < Λ ^{ de } (see above). If Δ(c, d, e) = 1, δ^{ cd } < δ^{ ce } and δ^{ cd } < δ^{ de } . This implies that even after projection from ℝ ^{ N } to ℝ ^{ n } , the classbased pairwise relationships within the data are accurately preserved (a classifier can be constructed which will correctly classify objects c, d, e in ℝ ^{ n } ).
Y will be a constant for a given set of objects C, and is based on forming unique triplets (c, d, e) where l(c) = l(d) = l(e) (triplets which are not class triplets). U = (Z  Y) will correspond to the number of class triplets that may be formed for set C. If all U class triplets have Δ ^{ l } (c, d, e) = 1, then it is possible to construct U classifiers which correctly classify the corresponding objects in these class triplets.
Definition 7 Given U unique class triplets (c, d, e) _{ l } ∈ C and an embedding ℝ ^{ n } of all objects c, d, e ∈ C, the associated classification accuracy${\varphi}^{Acc}\left({\mathbb{R}}^{n}\right)=\frac{{\sum}_{C}{\mathrm{\Delta}}^{l}\left(c,d,e\right)}{U}$
As illustrated previously, class triplets (c, d, e) _{ l } for which Δ ^{ l } (c, d, e) = 1 will correspond to those objects which will be classified correctly in ℝ ^{ n } . Therefore, the classification accuracy ϕ^{ Acc } (ℝ ^{ n } ) may simply be defined as the fraction of class triplets (c, d, e) _{ l } ∈ C for which Δ ^{ l } (c, d, e) = 1.
Theorem 1 For any ℝ ^{ n }, the corresponding ψ^{ ES } (ℝ ^{ n } ) increases monotonically as a function of ϕ^{ Acc } (ℝ ^{ n } ).
As $\frac{Y}{{\sum}_{C}{\mathrm{\Delta}}^{l}\left(c,d,e\right)}$ is a constant, ψ^{ ES } (ℝ ^{ n } ) increases monotonically with ϕ^{ Acc } (ℝ ^{ n } ). □
Thus an embedding ℝ ^{ n } with a high embedding strength will have a high classification accuracy. Practically, this implies that ψ^{ ES } (ℝ ^{ n } ) may be estimated via any measure of objectclass discrimination such as classification accuracy or clustervalidity measures. We have exploited this relationship in our algorithmic implementation (Section 2).
Endnotes
^{1} http://www.bic.mni.mcgill.ca/brainweb/
^{2}These datasets were downloaded from the Biomedical KentRidge Repositories at http://datam.i2r.astar.edu.sg/datasets/krbd/
Declarations
Acknowledgements
The authors would like to acknowledge Drs. Mark Rosen, John Tomasezewski, and Michael Feldman from the Hospital of the University of Pennsylvania for the use of ex vivo prostate MRI and histology data. They would also thank Andrew Janowczyk, Dr. Jonathan Chappelow, and Dr. James Monaco for results, discussions, and implementations used in this paper. This work was made possible via grants from the Department of Defense Prostate Cancer Research Program (W81XWH0810072), Wallace H. Coulter Foundation, National Cancer Institute (Grant Nos. R01CA13653501, R01CA14077201, and R03CA14399101), The Cancer Institute of New Jersey, and the Society for Imaging Informatics in Medicine.
Authors’ Affiliations
References
 Bellman R: Adaptive control processes: a guided tour. Princeton University Press; 1961.Google Scholar
 Jolliffe I: Principal Component Analysis. Springer; 2002.Google Scholar
 Lin T, Zha H: Riemannian Manifold Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 2008, 30(5):796–809.View ArticlePubMedGoogle Scholar
 Lee G, Rodriguez C, Madabhushi A: Investigating the Efficacy of Nonlinear Dimensionality Reduction Schemes in Classifying Gene and ProteinExpression Studies. IEEE/ACM Transactions on Computational Biology and Bioinformatics 2008, 5(3):1–17.View ArticleGoogle Scholar
 Saul L, Roweis S: Think globally, fit locally: unsupervised learning of low dimensional manifolds. Journal of Machine Learning Research 2003, 4: 119–155.Google Scholar
 Shi J, Malik J: Normalized Cuts and Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 2000, 22(8):888–905. 10.1109/34.868688View ArticleGoogle Scholar
 Tenenbaum J, Silva V, Langford J: A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science 2000, 290(5500):2319–2323. 10.1126/science.290.5500.2319View ArticlePubMedGoogle Scholar
 Dawson K, Rodriguez R, Malyj W: Sample phenotype clusters in highdensity oligonucleotide microarray data sets are revealed using Isomap, a nonlinear algorithm. BMC Bioinformatics 2005, 6(1):195. 10.1186/147121056195PubMed CentralView ArticlePubMedGoogle Scholar
 Madabhushi A, Shi J, Rosen M, Tomaszeweski JE, Feldman MD: Graph embedding to improve supervised classification and novel class detection: application to prostate cancer. Proc 8th Int'l Conf Medical Image Computing and ComputerAssisted Intervention (MICCAI) 2005, 729–37.Google Scholar
 Varini C, Degenhard A, Nattkemper T: Visual exploratory analysis of DCEMRI data in breast cancer by dimensional data reduction: A comparative study. Biomedical Signal Processing and Control 2006, 1(1):56–63. 10.1016/j.bspc.2006.05.001View ArticleGoogle Scholar
 Quinlan J: The effect of noise on concept learning. Morgan Kaufmann; 1986.Google Scholar
 Balasubramanian M, Schwartz EL, Tenenbaum JB, de Silva V, Langford JC: The Isomap Algorithm and Topological Stability. Science 2002, 295(5552):7a. 10.1126/science.295.5552.7aView ArticleGoogle Scholar
 Chang H, Yeung D: Robust locally linear embedding. Pattern Recognition 2006, 39(6):1053–1065. 10.1016/j.patcog.2005.07.011View ArticleGoogle Scholar
 Shao C, Huang H, Zhao L: PISOMAP: A New ISOMAPBased Data Visualization Algorithm with Less Sensitivity to the Neighborhood Size. Dianzi Xuebao(Acta Electronica Sinica) 2006, 34(8):1497–1501.Google Scholar
 Fred A, Jain A: Combining Multiple Clusterings Using Evidence Accumulation. IEEE Transactions on Pattern Analysis and Machine Intelligence 2005, 27(6):835–850. 10.1109/TPAMI.2005.113View ArticlePubMedGoogle Scholar
 Freund Y, Schapire R: A decisiontheoretic generalization of online learning and an application to boosting. Proc 2nd European Conf Computational Learning Theory 1995, 23–37.View ArticleGoogle Scholar
 Breiman L: Bagging predictors. Machine Learning 1996, 24(2):123–140.Google Scholar
 Dietterich T: Ensemble Methods in Machine Learning. Proc 1st Int'l Workshop on Multiple Classifier Systems 2000, 1–15.View ArticleGoogle Scholar
 MacQueen J: Some Methods for Classification and Analysis of MultiVariate Observations. Proc Fifth Berkeley Symposium on Mathematical Statistics and Probability 1967, 281–297.Google Scholar
 Fern X, Brodley C: Random Projection for High Dimensional Data Clustering: A Cluster Ensemble Approach. Proc 20th Int'l Conf Machine Learning 2003, 186–193.Google Scholar
 Venna J, Kaski S: Local multidimensional scaling. Neural Networks 2006, 19(6):889–899. 10.1016/j.neunet.2006.05.014View ArticlePubMedGoogle Scholar
 Samko O, Marshall A, Rosin P: Selection of the optimal parameter value for the Isomap algorithm. Pattern Recognition Letters 2006, 27(9):968–979. 10.1016/j.patrec.2005.11.017View ArticleGoogle Scholar
 Kouropteva O, Okun O, Pietikainen M: Selection of the Optimal Parameter Value for the Locally Linear Embedding Algorithm. Proc 1st Int'l Conf Fuzzy Systems and Knowledge Discovery 2002, 359–363.Google Scholar
 de Silva V, Tenenbaum J: Global Versus Local Methods in Nonlinear Dimensionality Reduction. Proc 15th Conf Adv Neural Information Processing Systems (NIPS) 2003, 705–712.Google Scholar
 ZelnikManor L, Perona P: Selftuning spectral clustering. Proc 17th Conf Adv Neural Information Processing Systems (NIPS) 2004, 1601–1608.Google Scholar
 Geng X, Zhan DC, Zhou ZH: Supervised nonlinear dimensionality reduction for visualization and classification. IEEE Transactions on Systems, Man, and Cybernetics: Part B, Cybernetics 2005, 35(6):1098–107. 10.1109/TSMCB.2005.850151View ArticleGoogle Scholar
 de Ridder D, Kouropteva O, Okun O, Pietikainen M, Duin R: Supervised Locally Linear Embedding. Proc Artificial Neural Networks and Neural Information Processing 2003, 333–341.Google Scholar
 Athitsos V, Alon J, Sclaroff S, Kollios G: BoostMap: An Embedding Method for Efficient Nearest Neighbor Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 2008, 30(1):89–104.View ArticlePubMedGoogle Scholar
 Lawrence N: Spectral Dimensionality Reduction via Maximum Entropy. Proc 14th Intn'l Conf Artificial Intelligence and Statistics (AISTATS) 2011, 51–59.Google Scholar
 Mao K, Liang F, Mukherjee S: Supervised Dimension Reduction Using Bayesian Mixture Modeling. Proc 13th Intn'l Conf Artificial Intelligence and Statistics (AISTATS) 2010, 501–508.Google Scholar
 Blum A, Mitchell T: Combining labeled and unlabeled data with cotraining. Proc 11th Annual Conf Computational Learning Theory 1998, 92–100.Google Scholar
 Hou C, Zhang C, Wu Y, Nie F: Multiple view semisupervised dimensionality reduction. Pattern Recognition 2009, 43(3):720–730.View ArticleGoogle Scholar
 Wachinger C, Yigitsoy M, Navab N: Manifold Learning for ImageBased Breathing Gating with Application to 4D Ultrasound. Proc 13th Int'l Conf Medical Image Computing and ComputerAssisted Intervention (MICCAI) 2010, 26–33.Google Scholar
 Wachinger C, Navab N: Manifold Learning for MultiModal Image Registration. Proc 11th British Machine Vision Conference (BMVC) 2010, 82.1–82.12.Google Scholar
 Cover T, Hart P: Nearest neighbor pattern classification. IEEE Transactions on Information Theory 1967, 13(1):21–27.View ArticleGoogle Scholar
 Tiwari P, Rosen M, Madabhushi A: Consensuslocally linear embedding (CLLE): application to prostate cancer detection on magnetic resonance spectroscopy. Proc 11th Int'l Conf Medical Image Computing and ComputerAssisted Intervention (MICCAI) 2008, 330–8.Google Scholar
 Ho TK: The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 1998, 20(8):832–844. 10.1109/34.709601View ArticleGoogle Scholar
 Levina E, Bickel P: Maximum Likelihood Estimation of Intrinsic Dimension. Proc 17th Conf Adv Neural Information Processing Systems (NIPS) 2005, 777–784.Google Scholar
 Kuncheva L: Combining pattern classifiers: methods and algorithms. WileyInterscience; 2004.View ArticleGoogle Scholar
 Halkidi M, Batistakis Y, Vazirgiannis M: On Clustering Validation Techniques. Journal of Intelligent Information Systems 2001, 17(2):107–145. 10.1023/A:1012801612483View ArticleGoogle Scholar
 Patel JK, Read CB: Handbook of the normal distribution. Marcel Dekker; 1996.Google Scholar
 Yang C, Duraiswami R, Gumerov NA, Davis L: Improved fast gauss transform and efficient kernel density estimation. Proc 9th IEEE Int'l Conf Computer Vision (ICCV) 2003, 664–671.View ArticleGoogle Scholar
 Comaniciu D, Meer P: Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002, 24(5):603–619. 10.1109/34.1000236View ArticleGoogle Scholar
 Haralick RM, Shanmugam K, Dinstein I: Textural Features for Image Classification. IEEE Transactions on Systems, Man and Cybernetics 1973, 3(6):610–621.View ArticleGoogle Scholar
 Madabhushi A, Feldman M, Metaxas D, Tomaszeweski J, Chute D: Automated Detection of Prostatic Adenocarcinoma from HighResolution Ex Vivo MRI. IEEE Transactions on Medical Imaging 2005, 24(12):1611–1625.View ArticlePubMedGoogle Scholar
 HerlidouMeme S, Constans JM, Carsin B, Olivie D, Eliat PA, NadalDesbarats L, Gondry C, Le Rumeur E, IdyPeretti I, de Certaines JD: MRI texture analysis on texture test objects, normal brain and intracranial tumors. Magnetic Resononance Imaging 2003, 21(9):989–993. 10.1016/S0730725X(03)002121View ArticleGoogle Scholar
 Dai JJ, Lieu L, Rocke D: Dimension reduction for classification with gene expression microarray data. Statistical Applications in Genetics and Molecular Biology 2006, 5(1):Article6.View ArticlePubMedGoogle Scholar
 CarballidoGamio J, Belongie S, Majumdar S: Normalized cuts in 3D for spinal MRI segmentation. IEEE Transactions on Medical Imaging 2004, 23(1):36–44. 10.1109/TMI.2003.819929View ArticlePubMedGoogle Scholar
 Kwan R, Evans A, Pike G: MRI simulationbased evaluation of imageprocessing and classification methods. IEEE Transactions on Medical Imaging 1999, 18(11):1085–1097. 10.1109/42.816072View ArticlePubMedGoogle Scholar
 Chappelow J, Bloch BN, Rofsky N, Genega E, Lenkinski R, DeWolf W, Madabhushi A: Elastic registration of multimodal prostate MRI and histology via multiattribute combined mutual information. Medical Physics 2011, 38(4):2005–2018. 10.1118/1.3560879PubMed CentralView ArticlePubMedGoogle Scholar
 Liu H, Li J, Wong L: A comparative study on feature selection and classification methods using gene expression profiles and proteomic patterns. Genome Informatics 2002, 13: 51–60.PubMedGoogle Scholar
 Zhao H: Combining labeled and unlabeled data with graph embedding. Neurocomputing 2006, 69(16):2385–2389. 10.1016/j.neucom.2006.02.010View ArticleGoogle Scholar
 Eisen M, Spellman P, Brown P, Botstein D: Cluster analysis and display of genomewide expression patterns. Proceedings of the National Academy of Sciences of the United States of America 1998, 95(25):14863–14868. 10.1073/pnas.95.25.14863PubMed CentralView ArticlePubMedGoogle Scholar
 Monaco JP, Tomaszewski JE, Feldman MD, Hagemann I, Moradi M, Mousavi P, Boag A, Davidson C, Abolmaesumi P, Madabhushi A: Highthroughput detection of prostate cancer in histological sections using probabilistic pairwise Markov models. Medical Image Analysis 2010, 14(4):617–629. 10.1016/j.media.2010.04.007PubMed CentralView ArticlePubMedGoogle Scholar
 Moakher M, Batchelor PG: Symmetric PositiveDefinite Matrices: From Geometry to Applications and Visualization. 2006.Google Scholar
 Bengio Y, Paiement J, Vincent P, Delalleau O, Le Roux N, Ouimet M: OutofSample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering. Proc 16th Conf Adv Neural Information Processing Systems (NIPS) 2004, 177–184.Google Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.