Support Vector Machine Implementations for Classification & Clustering

Background We describe Support Vector Machine (SVM) applications to classification and clustering of channel current data. SVMs are variational-calculus based methods that are constrained to have structural risk minimization (SRM), i.e., they provide noise tolerant solutions for pattern recognition. The SVM approach encapsulates a significant amount of model-fitting information in the choice of its kernel. In work thus far, novel, information-theoretic, kernels have been successfully employed for notably better performance over standard kernels. Currently there are two approaches for implementing multiclass SVMs. One is called external multi-class that arranges several binary classifiers as a decision tree such that they perform a single-class decision making function, with each leaf corresponding to a unique class. The second approach, namely internal-multiclass, involves solving a single optimization problem corresponding to the entire data set (with multiple hyperplanes). Results Each SVM approach encapsulates a significant amount of model-fitting information in its choice of kernel. In work thus far, novel, information-theoretic, kernels were successfully employed for notably better performance over standard kernels. Two SVM approaches to multiclass discrimination are described: (1) internal multiclass (with a single optimization), and (2) external multiclass (using an optimized decision tree). We describe benefits of the internal-SVM approach, along with further refinements to the internal-multiclass SVM algorithms that offer significant improvement in training time without sacrificing accuracy. In situations where the data isn't clearly separable, making for poor discrimination, signal clustering is used to provide robust and useful information – to this end, novel, SVM-based clustering methods are also described. As with the classification, there are Internal and External SVM Clustering algorithms, both of which are briefly described.

SVMs use variational methods in their construction and encapsulate a significant amount of discriminatory information in their choice of kernel. In reference [3] information-theoretic kernels provided notably better performance than standard kernels. Feature extraction was designed to arrive at probability vectors (i.e., discrete probability distributions) on a predefined, and complete, space of possibilities. (The different blockade levels, and their frequencies, the emission probabilities, and the transition probabilities, for example.) This turns out to be a very general formulation, wherein feature extraction makes use of signal decomposition into a complete set of separable states that can be interpreted or represented as a probability vector. A probability vector formulation also provides a straightforward hand-off to the SVM classifiers since all feature vectors have the same length with such an approach. What this means for the SVM, however, is that geometric notions of distance are no longer the best measure for comparing feature vectors. For probability vectors (i.e., discrete distributions), the best measures of similarity are the various information-theoretic divergences: Kullback-Leibler, Renyi, etc. By symmetrizing over the arguments of those divergences a rich source of kernels is A sketch of the hyperplane separability heuristic for SVM binary classification Figure 1 A sketch of the hyperplane separability heuristic for SVM binary classification. An SVM is trained to find an optimal hyperplane that separates positive and negative instances, while also constrained by structural risk minimization (SRM) criteria, which here manifests as the hyperplane having a thickness, or "margin," that is made as large as possible in seeking a separating hyperplane. A benefit of using SRM is much less complication due to overfitting (a common problem with Neural Network discrimination approaches). Given its geometric expression, it is not surprising that a key construct in the SVM formulation (via the choice of kernel) is the notion of "nearness" between instances (or nearness to the hyperplane, where it gives a measure of confidence in the classification, i.e., instances further from the decision hyperplane are called with greater confidence). Most notions of nearness explored in this context have stayed with the geometric paradigm and are known as "distance kernels," one example being the familiar Gaussian kernel which is based on the Euclidean distance: K Gaussian (x,y) = exp(-D Eucl. (x,y) 2 /2σ 2 ), where D Eucl. (x,y) = [∑ k (x k -y k )2] 1/2 is the usual Euclidean distance. Those kernels are used in the signal pattern recognition analysis in Figure 8 along with a new class of kernels, "divergence kernels," based on a notion of nearness appropriate when comparing probability distributions (or probability feature vectors). The main example of this is the Entropic Divergence Kernel: K Entropic = exp(-D Entropic. (x,y) 2 /2σ 2 ), where D Entropic. (x,y) = D(x||y) + D(y||x) and D(..||..) is the Kullback-Leibler Divergence (or relative entropy) between x and y.
obtained that works well with the types of probabilistic data obtained.
The SVM discriminators are trained by solving their KKT relations using the Sequential Minimal Optimization (SMO) procedure [4]. A chunking [5,6] variant of SMO also is employed to manage the large training task at each SVM node. The multi-class SVM training generally involves thousands of blockade signatures for each signal class. The data cleaning needed on the training data is accomplished by an extra SVM training round.

Binary Support Vector Machines
Binary Support Vector Machines (SVMs) are based on a decision-hyperplane heuristic that incorporates structural risk management by attempting to impose a traininginstance void, or "margin," around the decision hyperplane [1].
Feature vectors are denoted by x ik , where index i labels the M feature vectors (1 ≤ i ≤ M) and index k labels the N feature vector components (1 ≤ i ≤ N). For the binary SVM, labeling of training data is done using label variable y i = ± 1 (with sign according to whether the training instance was from the positive or negative class). For hyperplane separability, elements of the training set must satisfy the following conditions: w β x iβ -b ≥ +1 for i such that y i = +1, and w β x iβ -b ≤ -1 for y i = -1, for some values of the coefficients w 1 , ..., w N , and b (using the convention of implied sum on repeated Greek indices). This can be written more concisely as: y i (w β x iβ -b) -1 ≥ 0. Data points that satisfy the equality in the above are known as "support vectors" (or "active constraints").
Once training is complete, discrimination is based solely on position relative to the discriminating hyperplane: w β x iβ -b = 0. The boundary hyperplanes on the two classes of data are separated by a distance 2/w, known as the "margin," where w 2 = w β w β . By increasing the margin between the separated data as much as possible the optimal separating hyperplane is obtained. In the usual SVM formulation, the goal to maximize w -1 is restated as the goal to minimize w 2 . The Lagrangian variational formulation then selects an optimum defined at a saddle point of The saddle point is obtained by minimizing with respect to {w 1 , ...,w N ,b} and maximizing with respect to {α 1 , ..., α M }. If y i (w β x iβ -b) -1 ≥ 0, then maximization on α i is achieved for α i = 0. If y i (w β x iβ -b) -1 = 0, then there is no constraint on α i . If y i (w β x iβ -b) -1 < 0, there is a constraint violation, and α i → ∞. If absolute separability is possible the last case will eventually be eliminated for all α i , otherwise it's natural to limit the size of α i by some constant upper bound, i.e., max(α i ) = C, for all i. This is equivalent to another set of inequality constraints with α i ≤ C. Introducing sets of Lagrange multipliers, ξ γ and μ γ (1 ≤ γ ≤ M), to achieve this, the Lagrangian becomes: At the variational minimum on the {w 1 , ...,w N ,b} variables, w β = α γ y γ x γβ , and the Lagrangian simplifies to: L(α) = α 0 -(α δ y δ x δβ α γ y γ x γβ /2, with 0 ≤ α γ ≤ C (1 ≤ γ ≤ M) and α γ y γ = 0, where only the variations that maximize in terms of the α γ remain (known as the Wolfe Transformation). In this form the computational task can be greatly simplified. By introducing an expression for the discriminating hyperplane: f i = w β x iβ -b = α γ y γ x γβ x iβ -b, the variational solution for L(α) reduces to the following set of relations (known as the Karush-Kuhn-Tucker, or KKT, relations): (i) When the KKT relations are satisfied for all of the α γ (with α γ y γ = 0 maintained) the solution is achieved. (The constraint α γ y γ = 0 is satisfied for the initial choice of multipliers by setting the α's associated with the positive training instances to 1/N (+) and the α's associated with the negatives to 1/N (-) , where N (+) is the number of positives and N (-) is the number of negatives.) Once the Wolfe transformation is performed it is apparent that the training data (support vectors in particular, KKT class (ii) above) enter into the Lagrangian solely via the inner product x iβ x jβ . Likewise, the discriminator f i , and KKT relations, are also dependent on the data solely via the x iβ x jβ inner product.
Generalization of the SVM formulation to data-dependent inner products other than x iβ x jβ are possible and are usually formulated in terms of the family of symmetric positive definite functions (reproducing kernels) satisfying Mercer's conditions [1].
In terms of the new α 2 new, clipped , clipped as indicated above if necessary, the new α 1 becomes: where s ≡ y 1 y 2 as before. After the new α 1 and α 2 values are obtained there still remains the task of obtaining the new b value. If the new α 1 is not "clipped" then the update must satisfy the non-boundary KKT relation: y 1 f(x 1 ) = 1, i.e., f new (x 1 ) -y 1 = 0. By relating f new to f old the following update on b is obtained: b new1 = b -(f new (x 1 ) -y 1 ) -y 1 (α 1 new -α 1 old )K 11 -y 2 (α 2 new, clippedα 2 old )K 12 .
If α 1 is clipped but α 2 is not, the above argument holds for the α 2 multiplier and the new b is: If both α 1 and α 2 values are clipped then any of the b values between b new1 and b new2 is acceptable, and following the SMO convention, the new b is chosen to be:

Multiclass SVM Methods
The SVM binary discriminator offers high performance and is very robust in the presence of noise. This allows a variety of reductionist multiclass approaches, where each reduction is a binary classification (for classifying cards by suit, maybe classify as red or black first, then as heart or diamond for red and spade or club for black, for example). The SVM Decision Tree is one such approach, and a collection of them (a SVM Decision Forest) can be used to avoid problems with throughput biasing. Alternatively, the variational formalism can be modified to perform a multi-hyperplane optimization situation for a direct multiclass solution [7][8][9], and that is what is described next.

SVM-Internal Multiclass
In the formulation in [7], there are 'k' classes and hence 'k' linear decision functions -a description of their approach is given here. For a given input 'x', the output vector corresponds to the output from each of these decision functions. The class of the largest element of the output vector gives the class of 'x'.
Each decision function is given by: f m (x) = w m .x + b m for all m = (1,2, ..., k). If y i is the class of the input x i , then for each input data point, the misclassification error is defined as follows: max m {f m (x i ) + 1 -δ i m } -f yi (x i ), where δ i m is 1 if m = y i and 0 if m ≠ y i . We add the slack variable ζ i where ζ i ≥ 0 for all i that is proportional to the misclas- To minimize this classification error and maximize the distance between the hyper-planes (Structural Risk Minimization) we have the following formulation: where β > 0 is defined as a regularization constant.
Constraint: w yi .x i + b yi -w m .x i -b m -1 + ζ i + δ i m ≥ 0 for all i,m Note: the term (1/2)∑ m b m 2 is added for de-coupling, 1/β = C, and m = y i in the above constraint is consistent with ζ i ≥ 0. The Lagrangian is: Where K ij = x i .x j is the Kernel generalization. In vector notation: Let τ i = Δ yi -A i . Hence after ignoring the constant: -1/ 2∑ i,j τ i .τ j (K ij + β) + β∑ i Δ; yi τ i , subject to: τ i ≤ Δ yi , τ i .1 = 0, i = 1 ...l. The dual is solved (determine the optimum values of all the τs) using the decomposition method.
The Lagrangian of the dual is: We take the gradient of the Lagrangian with respect to τ i m :

Decomposition Method to Solve the Dual
Using the method in [7] to solve the Dual, maximize Expanding in terms of a single 'τ' vector: Where: Therefore ignoring the constant term 'C p ', we have to minimize: Subject to: τ p ≤ Δ yp and τ p .1 = 0 The above equation can also be written as: The Lagrangian is given by: Hence by combining the above in-equalities, we have: v m = Min{D m , σ}, or ∑ m v m = ∑ m Min{D m , σ} = ∑ m D m -1. The above equation uniquely defines the 'σ' that satisfies the above equation AND that 'σ' is the optimal solution of the quadratic optimization problem. (Refer to [7] for a formal proof).

SVM-Internal Clustering
Let {x i } be a data set of 'N' points in R d . Using a non-linear transformation φ, we transform 'x' to some high-dimensional space called Kernel space and look for the smallest enclosing sphere of radius 'R'. Hence we have: ||φ(x j ) -a || 2 ≤ R 2 for all j = 1,...,N; where 'a' is the center of the sphere.
Soft constraints are incorporated by adding slack variables 'ζ j ': We introduce the Lagrangian as: where C is the cost for outliers and hence C∑ j ζ j is a penalty term. Setting to zero the derivative of 'L' w.r.t. R, a and ζ we have: ∑ j β j = 1; a = ∑ j β j φ(x j ); and β j = C -μ j .
In the kernel space of a data point 'x j ' if ζ j > 0, then β j = C and hence it lies outside of the sphere i.e. R 2 < ||φ(x j ) -a || 2 . This point becomes a bounded support vector or BSV. Similarly if ζ j = 0, and 0 < β j < C, then it lies on the surface of the sphere i.e. R 2 = ||φ(x j ) -a || 2 . This point becomes a support vector or SV. If ζ j = 0, and β j = 0, then R 2 > ||φ(x j ) -a || 2 and hence this point is enclosed with-in the sphere.

Nanopore Detector based Channel Current Cheminformatics
All data analyzed is obtained from a nanopore detector and relates to single molecule blockades of a single protein channel. The protein channel is the α-hemolysin pore-forming toxin from Staphylococcus aureus, which has a molecule-sized channel opening for partial capture, if not translocation, of biomolecules drawn in by electrophoretic forces (such as DNA) [3,[10][11][12][13][14][15][16][17][18][19][20]. Further details on the detector and signal processing architecture are shown in Fig. 2. Further detail on the components of the extracted SVM feature vectors (on events due to individual blockade events), are given in the Methods. Although the figure can only show one SVM classifier implementation (that used in [3]), the data sets examined by all the SVMs described are kept the same (for comparative purposes), so the signal acquisition and feature extraction stages show how the SVM feature vectors are obtained.

Information measures
The fundamental information measures are Shannon entropy, mutual information, and relative entropy (also known as the Kullback-Leibler divergence or distance). Shannon entropy, σ = -Σ x p(x)log(p(x)), is a measure of the information in distribution p(x). Mutual Information, μ = Σ x Σ y p(xy)log(p(xy)/p(x)p(y)), is a measure of information one random variable has about another random variable. Relative Entropy (Kullback-Leibler distance): ρ = Σ x p(x) log(p(x)/q(x)), is a measure of distance between two probability distributions. Mutual information is a special case of relative entropy between a joint probability (two-component in simplest form) and the product of component probabilities.

Khinchin derivation of Shannon entropy
In his now famous 1948 paper, Claude Shannon [21] provided a qualitative measure for entropy in connection with communication theory. The Shannon entropy measure was later put on a more formal footing by A. I. Khinchin in an article where he proves that with certain reasonable assumptions the Shannon entropy is unique [22]. A statement of the theorem is as follows:

Khinchine Uniqueness Theorem
Let H(p 1 ,p 2 ,...,p n ) be a function defined for any integer n and for all values p 1 ,p 2 ,...,p n such that p k ≥0 (k = 1,2,...,n), and Σ k p k = 1. If for any function n this function is continuous with respect to its arguments, and if the function obeys the three properties listed below, then H(p 1 ,p 2 ,...,p n ) = -λΣ k p k log(p k ), where λ is a positive constant (with Shannon entropy recovered for convention λ = 1). The three properties are: (1) For given n and for Σ k p k = 1, the function takes its largest value for p k = 1/n (k = 1,2,...,n). This is equivalent to Laplace's principle of insufficient reason, which says if you don't know anything assume the uniform distribution (also agrees with Occam's Razor assumption of minimum structure).
(2) H(ab) = H(a) + H a (b), where H a (b) = -Σ a p(a)log(p(b|a)), is the conditional entropy. This is consistent with H(ab)=H(a)+H(b), for probabilities of a and b independent, with modifications involving conditional probability being used when not independent.

Relative Entropy Uniqueness
This falls out of a geometric formalism on families of distributions: the Information Geometry formalism described by S. Amari [23][24][25]. Together with Laplace's principle of insufficient reason on the choice of "reference" distribution in the relative entropy expression, this will reduce to Shannon entropy, and thus uniqueness on Shannon entropy from a geometric context. The parallel with geometry is the Euclidean distance for "flat" geometry (simplest assumption of structure), vs. the "distance" between distributions as described by the Kullback-Leibler divergence.
a. (A) shows a nanopore device based on the α-hemolysin channel Figure 2 a. (A) shows a nanopore device based on the α-hemolysin channel. It has been used for analysis of single DNA molecules, such as ssDNA, shown, and dsDNA, a nine base-pair DNA hairpin is shown in (B) superimposed on the channel geometry. The channel current blockade trace for the nine base-pair DNA hairpin blockade from (B) is shown in (C). b shows the signal processing architecture that was used to classify DNA hairpins with this approach: Signal acquisition was performed using a time-domain, thresholding, Finite State Automaton, followed by adaptive pre-filtering using a wavelet-domain Finite State Automaton. Hidden Markov Model processing with Expectation-Maximization was used for feature extraction on acquired channel blockades. Classification was then done by Support Vector Machine on five DNA molecules: four DNA hairpin molecules with nine base-pair stem lengths that only differed in their blunt-ended DNA termini, and an eight base-pair DNA hairpin. The accuracy shown is obtained upon completing the 15 th single molecule sampling/classification (in approx. 6 seconds), where SVM-based rejection on noisy signals was employed.
(page number not for citation purposes)

The Success of Distributions of Nature suggests Generalization from Geometric Feature-Space Kernels to Distribution Feature-Space Kernels
Using the Shannon entropy measure it is possible to derive the classic probability distributions of statistical physics by maximizing the Shannon measure subject to appropriate linear momentum constraints. Constrained variational optimizations involving the Shannon entropy measure can, thus, provide a unified framework with which to describe all, or most, of statistical mechanics. The distributions derivable within the maximum entropy formalism include the Maxwell-Boltzmann, Bose-Einstein, Fermi-Dirac, and Intermediate distributions. The maximum entropy method for defining statistical mechanical systems has been extensively studied by [26].
Both statistical estimation and maximum entropy estimation are concerned with drawing inferences from partial information. The maximum entropy approach estimates a probability density function when only a few moments are known (where there are an infinite number of higher moments). The statistical approach estimates the density function when only one random sample is available out of an infinity of possible samples. The maximum entropy estimation may be significantly more robust (against over-fitting, for example) in that it has an Occam's Razor argument that "cuts both ways" -use all of the information given and avoid using any information not given. This means that out of all of the probability distributions consistent with the set of constraints, choose the one that has maximum uncertainty, i.e., maximum entropy [27].
At the same time that Jaynes was doing his work, essentially an optimization principle based on Shannon entropy, Soloman Kullback was exploring optimizations involving a notion of probabilistic distance known as the Kullback-Leibler distance, referred to above as the relative entropy [28]. The resulting minimum relative entropy (MRE) formalism reduces to the maximum entropy formalism of Jaynes when the reference distribution is uniform. The information distance that Kullback and Leibler defined was an oriented measure of "distance" between two probability distributions. The MRE formalism can be understood to be an extension of Laplace's Principle of Insufficient Reason (e.g., if nothing known assume the uniform distribution) in a manner like that employed by Khinchine in his uniqueness proof, but now incorporating constraints.
In their book Entropy Optimization Principles with Applications [27], Kapur and Kesavan argue for a generalized entropy optimization approach to the description of distributions. They believe every probability distribution, theoretical or observed, is an entropy optimization distribution, i.e., it can be obtained by maximizing an appro-priate entropy measure, or by minimizing a relative entropy measure with respect to an appropriate a priori distribution. The primary objective in such a modeling procedure is to represent the problem as a simple combination of probabilistic entities that have a simple set of moment constraints. Generalized measures of distributional distance can also be explored along the lines of generalized measures of geometric distance. In physics, not every geometric distance is of interest, however, since the special theory of relativity tells us that spacetime is locally flat (Lorentzian, which is Euclidean on spatial slices), with metric generalization the Riemannian metrics. Likewise, perhaps not all distributional distance measures are created equal either. What the formalism of Information Geometry [23][24][25] reveals, among other things, is that relative entropy is uniquely structureless (like flat geometry) and is perturbatively stable, i.e., has a well-defined Taylor expansion at short divergence range, just like the locally Euclidean metrics at short distance range.

SVM Kernel/Algorithm Variants
The SVM Kernels of interest are "regularized" distances or divergences, where they are regularized if in the form of an exponential with argument the negative of some distancemeasure squared (d 2 (x,y)) or symmetrized divergence measure (D(x,y)), the former if using a geometric heuristic for comparison of feature vectors, the latter if using a distributional heuristic. For the Gaussian Kernel: d 2 (x,y) = Σk(xk-yk) 2 ; for the Absdiff Kernel d 2 (x,y)=(Σk|xk-yk|) 1/2 ; and for the Symmetrized Relative Entropy Kernel D(x,y)= D(x||y)+D(y||x), where D(x||y) is the standard relative entropy. Results are shown in Fig. 3.
The SVM algorithm variants being explored are only briefly mentioned here. In the standard Platt SMO algorithm, η = 2*K12-K11-K22, and speedup variations are described to avoid calculation of this value entirely. A middle ground is sought with the following definition "η = 2*K12-2; If (η >= 0) { η = -1;}" (labeled WH SMO in Fig.  3, underflow handling and other implementations differ slightly in the implementation shown as well). Fig. 4 shows the percent increase in iterations-to-convergence against the 'C' value. Fig. 5 shows the number of bounded support vectors (BSV) as a function of 'C' value.

SVM-Internal Speedup via differentiating BSVs and SVs
Since the algorithm presented in [7] does not differentiate between SV and BSV, a lot of time is spent in trying to adjust the weights of the BSV i.e. weak data. The weight of a BSV may range from [0, 0.5) in their algorithm. In our modification to the algorithm, shown below, as soon as we identify the BSV (as specified by Case III conditions), its weight is no longer adjusted. Hence faster convergence is achieved without sacrificing accuracy:

Implementation/Kernel Label Accuracy (SN+SP)/2*100%
The percent increase in iterations-to-convergence against the 'C' value Figure 4 The percent increase in iterations-to-convergence against the 'C' value. For very low values of 'C' the gain is doubled while for very large values of 'C' the gain is low (almost constant for C > 150). Thus we note the dependence of the gain on 'C' value.

(page number not for citation purposes)
Implies ζ i > 0 Hence f yi -f m max -1 + ζ i = 0 Or f yi -f m max -1 < 0

Data Rejection Tuning with SVM-Internal vs SVM-External Classifiers
The SVM Decision Tree shown in Fig. 2b obtained nearly perfect sensitivity and specificity, with a high data rejection rate, and a highly non-uniform class signal-calling throughput. In Fig. 6, the Percentage Data Rejection vs SN+SP curves are shown for test data classification runs with a binary classifier with one molecule (the positive, given by label) versus the rest (the negative). Since the signal calling wasn't passed through a Decision Tree, the way these curves were generated, they don't accurately reflect total throughput, and they don't benefit from the "shielding" shown in the Decision Tree in Fig. 2b prototype. In the SVM Decision Tree implementation described in Fig. 2b [3], this is managed more comprehensively, to arrive at a five-way signal-calling throughput at the furthest node of 16% (in Fig. 1a, 9CG and 9AT have to pass to the furthest node to be classified), while the best throughput, for signal calling on the 8GC molecules, is 75%.
The SVM Decision Tree classifier's high, non-uniform, rejection can be managed by generalizing to a collection of Decision Trees (with different species at the furthest node). The problem is that tuning and optimizing a single decision tree is already a large task, even for five species (as in Fig. 2). With a collection of trees this problem is seemingly compounded, but can actually be lessened in some ways in that now each individual tree need not be so welltuned/optimized. Although more complicated to implement than an SVM-External method, the SVM-Internal multiclass methods are not similarly fraught with tuning/ optimization complications. Fig. 7 shows the Percentage Data Rejection vs SN+SP curves on the same train/test data splits as used for Fig. 6, except now the drop curves are to be understood as simultaneous curves (not sequential application of such curves as in Fig. 6). Thus, comparable, or better, performance is obtained with the multiclass-internal approach and with far less effort since there is no managing and tuning of Decision Trees. Another surprise, and even stronger argument for the SVM-Internal approach to the problem, is that a natural drop zone is indicated by the margin.  Table 1. Using the margin drop approach, there is even less tuning, and

Marginal Drop with SVM-Internal
The Percentage Data Rejection vs SN+SP curves are shown for test data classification runs with a multiclass discriminator Figure 7 The Percentage Data Rejection vs SN+SP curves are shown for test data classification runs with a multiclass discriminator. The following criterion is used for dropping weak data: for any data point x i ; if max m {f m (x i )} ≤ Confidence Parameter, then the data point x i is dropped. For this data set using AbsDiff kernel (σ 2 = 0.2) performed best, and a confidence parameter of 0.8 achieve 100% accuracy.
The Percentage Data Rejection vs SN+SP curves are shown for test data classification runs with a binary classifier with one molecule (the positive, given by label) versus the rest (the negative) Figure 6 The Percentage Data Rejection vs SN+SP curves are shown for test data classification runs with a binary classifier with one molecule (the positive, given by label) versus the rest (the negative). Since the signal calling wasn't passed through a Decision Tree, it doesn't accurately reflect total throughput, and they don't benefit from the "shielding" shown in the Decision Tree in Fig. 1 prototype. The Relative Entropy Kernel is shown because it provided the best results (over Gaussian and Absdiff).
there is improved throughput (approximately 75% for all species).

SVM-Internal Clustering
The SVM-Internal approach to clustering was originally defined by [29]. Data points are mapped by means of a kernel to a high dimensional feature space where we search for the minimal enclosing sphere. In what follows, Keerthi's method is used to solve the dual (see Methods for further details).
The minimal enclosing sphere, when mapped back into the data space, can separate into several components; each enclosing a separate cluster of points. The width of the kernel (say Gaussian) controls the scale at which the data is probed while the soft margin constant helps to handle outliers and over-lapping clusters. The structure of a dataset is explored by varying these two parameters, maintaining a minimal number of support vectors to assure smooth cluster boundaries.
We have used the algorithm defined in [29] to identify the clusters, with methods adapted from [30,31 for their handling. If the number of data points is 'n', then we require n(n-1)/2 number of comparisons. We have made modifications to the algorithm such that we eliminate comparisons that do not have an impact on the cluster connectivity. Hence the number of comparisons required will be less than n(n-1)/2.  Tables 2 and 3. The approach to the solving the Dual problem is shown in the Methods.

SVM-External Clustering
As with the multiclass SVM discriminator implementations, the strong performance of the binary SVM enables SVM-External as well as SVM-Internal approaches to clustering. Our external-SVM clustering algorithm clusters data vectors with no a priori knowledge of each vector's class. The algorithm works by first running a Binary SVM against a data set, with each vector in the set randomly labeled, until the SVM converges (Fig. 8). In order to obtain convergence, an acceptable number of KKT violators must be found. This is done through running the SVM on the randomly labeled data with different numbers of allowed violators until the number of violators allowed is near the lower bound of violators needed for the SVM to converge on the particular data set. Choice of an appropriate kernel and an acceptable sigma value also will affect convergence. After the initial convergence is achieved, the sensitivity + specificity will be low, likely near 1. The algorithm now improves this result by iteratively relabeling the worst misclassified vectors, which have confidence factor values beyond some threshold, followed by rerunning the SVM on the newly relabeled data set. This continues until no more progress can be made. Progress is determined by an increasing value of sensitivity + specificity, hopefully nearly reaching 2. After this process, a high percentage of the previously unknown class labels of the data set will be known. With sub-cluster identification upon iterating the overall algorithm on the positive and negative clusters identified (until the clusters are no longer separable into sub-clusters), this method provides a way to cluster data sets without prior knowledge of the data's clustering characteristics, or the number of clusters. Figures 9 and 10 show clustering runs on a data set with a mixture of 8GC and 9GC DNA hairpin data. The set consists of 400 elements. Half of the elements belong to each class. The SVM uses a Gaussian Kernel and allows 3% KKT Violators.

Machine Learning and Cheminformatics Tools are Accessible via Website
The web-site provides an interface to several binary SVM variants (with other novel kernel selections), to a multiclass (internal) SVM, an FSA-based nanopore spike detector, and an HMM-based channel current feature extraction. New, web-accessible, channel current analysis tools, have also been developed for kinetic feature extraction (via channel current sub-level lifetimes), and clustering. The website is designed using HTML and CGI scripts that are executed to process the data sent when a form filled in by the user is received at the web server -results are then e-mailed to the address indicated by the user. The interface to this and all other software described is available via the group Home Page: http://logos.cs.uno.edu/ nano/ (see Fig. 11). The SVM interface offers options on chunk processing for large training sets (SV-carry by appending to next training chunk and SV-carry by maintaining state and injecting ("unfreezing") the next training chunk (a specialized α-heuristic). The interface offers use of arbitrary or structured feature vectors -where structured, in this case, corresponds to feature vector components that satisfy the properties of a non-trivial, nonreducible, discrete probability distribution. There is an SVM interface for a new single-optimization multiclass SVM discriminator (it simultaneously optimizes multiple hyperplanes). There is also an interface for our SVM-based clustering methods.

Adaptive Feature Extraction/Discrimination
Adaptive feature extraction and discrimination, in the context of SVMs, can be accomplished by small batch reprocessing using the learned support vectors together with the new information to be learned. The benefit is that the easily deployed properties of SVMs can be retained while at the same time co-opting some of the on-line adaptive characteristics familiar from on-line learning with neural nets. This is also compatible with the chunking processing that is already implemented. A situation where such adaptation might prove necessary in nanopore signal analysis is if the instrumentation was found to have measurable, but steady, drift (at a new level of sensitivity for example). At the forefront of online adaptation, where the discrimination and feature extraction optimizations are inextricably mixed, further progress may derive  benefit from the Information-Geometrical methods of S. Amari [23][24][25].

Robust SVM performance in the presence of noise
In a parallel datarun to that indicated in Fig. 2a

AdaBoost Feature Selection
If SVM performance on the full HMM parameter set (the features extracted for each blockade signal) offers equiva-lent performance after rejecting weak data, then the possibility for significant improvement with selection on good parameters. An AdaBoost method is being used to select HMM parameters by representing each feature vector component as an independent Naïve Bayes classifier (trained on the data given), that then comprise the pool of experts in the AdaBoost algorithm [32][33][34]. The experts AdaBoost assigns heaviest weighting will then the components selected in the new, AdaBoost assigned, feature vector compression.

Conclusion
• External Multi-class SVM gave best results with Sentropic Kernel while Internal Multi-class SVM gave best results with AbsDiff kernel.
• Internal Multi-class approach overcomes the need to search for the best performing tree out of many possibili-Shown is the schematic for an "external" SVM clustering algorithm Figure 8 Shown is the schematic for an "external" SVM clustering algorithm. (a) The percentage correct classification (an indication of the clustering success) is shown with successive iteration of the clus-tering algorithm Figure 9 (a) The percentage correct classification (an indication of the clustering success) is shown with successive iteration of the clustering algorithm. Five separate test runs are shown, on different data from the same classes. Note that the plateau at around 0.9, this is approximately the performance of a supervised binary SVM on the same data (i.e., perfect separation isn't possible with this data without employing weak-data rejection). (b) The degradation in clustering performance for less optimal selection of kernel and tuning parameter (variance in case of Gaussian). (c) The degradation in clustering performance for non-optimal selection of kernel and tuning parameter (variance in case of Gaussian). (d) Summary of the degradation in clustering performance for less optimal selection of kernel and tuning parameter -with averages of the five test-runs are used as representative curves for that kernel/tuning selection in the above. ties. This is a huge advantage especially when the number of classes is large.
• Using a margin to define the drop zone for the internal multi-class approach produced far better results i.e. fewer data were dropped to achieve 100% accuracy.
• Additional benefit of using the margin is that the drop zone tuning to achieve 100% accuracy becomes trivial.
• External and Internal SVM Clustering Methods were also examined. The results show that our SVM-based clustering implementations can separate data into proper clusters without any prior knowledge of the elements' classification. this can be a powerful resource for insight into data linkages (topology).

The Feature Extraction used to obtain the Feature Vectors for SVM analysis Signal Preprocessing Details
The Nanopore Detector is operated such that a stream of 100 ms samplings are obtained (throughput was approximately one sampling per 300 ms in [3]). Each 100 ms signal acquired by the time-domain FSA consists of a sequence of 5000 sub-blockade levels (with the 20 μs analog-to-digital sampling). Signal preprocessing is then used for adaptive low-pass filtering. For the data sets examined, the preprocessing is expected to permit compression on the sample sequence from 5000 to 625 samples (later HMM processing then only required construction of a dynamic programming table with 625 columns). The signal preprocessing makes use of an off-line wavelet station-arity analysis (Off-line Wavelet Stationarity Analysis, Figure 2b, also see [35]).

HMMs and Supervised Feature Extraction Details
With completion of preprocessing, an HMM [36] is used to remove noise from the acquired signals, and to extract features from them (Feature Extraction Stage, Fig. 2b). The HMM is, initially, implemented with fifty states, corresponding to current blockades in 1% increments ranging from 20% residual current to 69% residual current. The HMM states, numbered 0 to 49, corresponded to the 50 different current blockade levels in the sequences that are processed. The state emission parameters of the HMM are initially set so that the state j, 0 <= j <= 49 corresponding to level L = j+20, can emit all possible levels, with the probability distribution over emitted levels set to a discretized Gaussian with mean L and unit variance. All transitions between states are possible, and initially are equally likely. Each blockade signature is de-noised by 5 rounds of Expectation-Maximization (EM) training on the parameters of the HMM. After the EM iterations, 150 parameters are extracted from the HMM. The 150 feature vector components are extracted from the 50 parameterized emission probabilities, a 50-element compressed representation of the 50 2 transition probabilities, and an a posteriori information from the Viterbi path solution which is, essentially, a de-noised histogram of the bloackade sub-level occupation probabilities (further details in [3]). This information elucidates the blockade levels (states) characteristic of a given molecule, and the occupation probabilities for those levels, but doesn't directly provide kinetic information. An HMM-with-Duration has recently been introduced to better capture the latter information, but such feature vectors are not used in the studies shown in this paper, so this approach isn't discussed further in this paper.

Solving the Dual (Based on Keerthi's SMO [37])
The dual formalism is: 1 -∑ i,j β i β j K ij where 0 ≤ β i ≤ C; K ij = exp(-||x i -x j || 2 /2σ 2 ), also ∑ i β i = 1. For any data point 'x k ', the distance of its image in kernel space from the center of the sphere is given by: R 2 (x k ) = 1 -2∑ i β i K ik + ∑ i,j β i β j K ij . The radius of the sphere is R = {R(x k ) | x k is a Support Vectors}, hence data points which are Support Vectors lie on cluster boundaries. Outliers are points that lie outside of the sphere and therefore they do not belong to any cluster i.e. they are Bounded Support Vectors. All other points are enclosed by the sphere and therefore they lie inside their respective cluster. KKT Violators are given as: (i) If 0 < β i < C and R(x i ) ≠ R; (ii) If β i = 0 and R(x i ) > R; and (iii) If β i = C and R(x i ) < R.

Convergence Time for Clustering Solution
Several channel current cheminformatics tools are available for use via web interfaces at http://logos.cs.uno.edu/~nano/ Figure 11 Several channel current cheminformatics tools are available for use via web interfaces at http://logos.cs.uno.edu/~nano/. These tools include a variety of SVM interfaces for classification and clustering (binary and multiclass), and HMM tools for feature extraction and structure identification (with applications to both channel current cheminformatics and computational genomics).
Divide data into three sets: Set I if 0 < β i < C; Set II if β i = 0; and Set III if β i = C.
In every iteration execute the following two paths alternatively until there are no KKT violators: Set II and R 2 (x i ) > R 2 _up; choose R 2 _up for joint optimization Set III and R 2 (x i ) < R 2 _low; choose R 2 _low for joint optimization Set I and R 2 (x i ) > R 2 _up + 2*tol OR R 2 (x i ) < R 2 _low -2*tol; choose R 2 _low or R 2 _up for joint optimization depending on which gives a worse

The SVM-External Clustering Method
The SVM-clustering software is written in Perl. It runs data on a separate Binary SVM also written in Perl. This SVM uses a C file for kernel calculations. The data run on the SVM is created by running raw data through a tFSA/ HMM(written in C), which creates a data set that contains 151 feature vectors for each element. The following is a simple step-by-step description of the basic algorithm used for SVM-clustering on this data: 1. Start with a set of data vectors (obtained through running raw data through tFSA/HMM feature extraction in Fig. 2b).
2. Randomly label each vector in the set as positive or negative.
3. Run the SVM on the randomly labeled data set until convergence is obtained (random relabeling is needed if prior random label scheme does not allow for convergence).
4. After initial convergence is obtained for the randomly labeled data set, relabel the misclassified data vectors, which have confidence factor values greater than some threshold.
5. Rerun the SVM on the newly relabeled data set.
6. Continue relabeling and rerunning SVM until no vectors in the data set are misclassified (or there is no improvement).
Publish with Bio Med Central and every scientist can read your work free of charge