 Research article
 Open access
 Published:
Recursive gene selection based on maximum margin criterion: a comparison with SVMRFE
BMC Bioinformatics volumeÂ 7, ArticleÂ number:Â 543 (2006)
Abstract
Background:
In class prediction problems using microarray data, gene selection is essential to improve the prediction accuracy and to identify potential marker genes for a disease. Among numerous existing methods for gene selection, support vector machinebased recursive feature elimination (SVMRFE) has become one of the leading methods and is being widely used. The SVMbased approach performs gene selection using the weight vector of the hyperplane constructed by the samples on the margin. However, the performance can be easily affected by noise and outliers, when it is applied to noisy, small sample size microarray data.
Results:
In this paper, we propose a recursive gene selection method using the discriminant vector of the maximum margin criterion (MMC), which is a variant of classical linear discriminant analysis (LDA). To overcome the computational drawback of classical LDA and the problem of high dimensionality, we present efficient and stable algorithms for MMCbased RFE (MMCRFE). The MMCRFE algorithms naturally extend to multiclass cases. The performance of MMCRFE was extensively compared with that of SVMRFE using nine cancer microarray datasets, including four multiclass datasets.
Conclusion:
Our extensive comparison has demonstrated that for binaryclass datasets MMCRFE tends to show intermediate performance between hardmargin SVMRFE and SVMRFE with a properly chosen softmargin parameter. Notably, MMCRFE achieves significantly better performance with a smaller number of genes than SVMRFE for multiclass datasets. The results suggest that MMCRFE is less sensitive to noise and outliers due to the use of average margin, and thus may be useful for biomarker discovery from noisy data.
Background
Microarray technology allows us to measure the expression levels of thousands of genes simultaneously. A vast amount of data produced by microarrays pose a great challenge on conventional data mining and machine learning methods, because the number of genes often exceeds tens of thousands, whereas the number of samples is at most a few hundred.
Along with clustering and classification of genes and/or samples, gene selection is an important aspect of microarray data analysis, and has been a central issue in recent years [1, 2]. Specifically, gene selection is used to identify genes most relevant to sample classification, for example, those differentiate between normal and cancerous tissue samples. Gene selection plays essential roles in classification tasks. It improves the prediction accuracy of classifiers by using only discriminative genes. It also saves computational costs by reducing dimensionality. More importantly, if it is possible to identify a small subset of biologically relevant genes, it may provide insights into understanding the underlying mechanism of a specific biological phenomenon. Also, such information can be useful for designing less expensive experiments by targeting only a handful of genes.
The most common gene selection approach is socalled gene ranking. It is a univariate approach in the sense that each gene is evaluated individually with respect to a certain criterion that represents class discrimination ability. The criteria often used are e.g., tstatistics, the signaltonoise (S2N) ratio [3, 4] and the betweengroup to withingroup (BW) ratio [5]. Although such gene ranking criteria are simple to use, they ignore correlations or interactions among genes, which may be essential to class discrimination and characterization.
Among existing gene selection methods, support vector machinebased recursive feature elimination (SVMRFE) [6] has become one of the leading methods and is being widely used. It is a multivariate approach, hence the correlations among genes can be taken into account. Moreover, since the selection is based on an SVM classifier, a subset of genes that yields high classification performance can be identified. Recently, the successful application of SVMRFE has motivated the development of several SVMbased gene selection methods [7â€“9]. The SVMbased approach performs gene selection using the weight vector of the hyperplane constructed by the samples on the margin, i.e. support vectors. However, while this property may be crucial for achieving good generalization performance, the effect of using support vectors on gene selection remains unclear, especially when it is applied to noisy, small sample size microarray data. A recent work by Li and Yang [10] implies that only penalizing redundant genes for the samples on the margin may lead to poorer performance.
In this paper, we propose a recursive gene selection method based on the maximum margin criterion (MMC) [11], which is a variant of classical linear discriminant analysis (LDA). Guyon et al. [6] compared the performance between SVMRFE and classical LDAbased RFE (LDARFE), and claimed that the use of support vectors is critical in eliminating irrelevant genes. However, the comparison is insufficient in the following respects:

For computational reasons, LDARFE was performed by eliminating half of genes at each iteration, whereas SVMRFE by eliminating one gene at a time.

Crossvalidation was performed improperly [12].

The comparison was made only on a single dataset.
The computational drawback of classical LDA limits the use of LDARFE for gene selection. This paper presents efficient and stable algorithms for MMCbased RFE (MMCRFE), which overcomes the singularity problem of classical LDA and the problem of high dimensionality. To validate the effectiveness of MMCRFE, we extensively compare its performance with that of SVMRFE using nine cancer microarray datasets.
Results and discussion
Datasets
In this study, we used nine public datasets of cancer microarrays. Five of the datasets concern binaryclass prediction problems: normal versus tumor for Colon cancer [13] and Prostate cancer [14], ALL versus AML for Leukemia [3], and clinical outcome for Medulloblastoma [15] and Breast cancer [16]. Four of the datasets are on multiclass subtype prediction problems: MLL [17], SRBCT [18], CNS [15], and NCI60 [19]. The details of these datasets are described below:
Colon cancer dataset [13]
This Affymetrix highdensity oligonucleotide array dataset contains 62 samples from 2 classes of coloncancer patients: 40 normal healthy samples and 22 tumor samples. The expression profiles of 2000 genes are used. The dataset is publicly available at [20].
Prostate cancer dataset [14]
This Affymetrix highdensity oligonucleotide array dataset contains 102 samples from 2 classes: 50 normal tissue samples and 52 prostate tumor samples. The expression profiles of 12600 genes are used. The dataset is publicly available at [21].
Leukemia dataset [3]
This Affymetrix highdensity oligonucleotide array dataset contains 38 samples from 2 classes of leukemia: 27 acute lymphoblastic leukemia (ALL) and 11 acute myeloid leukemia (AML). The expression profiles of 7129 genes are used. The dataset is publicly available at [21]. Other 34 samples consisting of 20 ALL and 14 AML are used as an independent test set as mentioned later.
Medulloblastoma dataset [15]
This Affymetrix highdensity oligonucleotide array dataset contains 60 samples from 2 classes on patient survival with medulloblastoma: 21 treatment failures and 39 survivors. The expression profiles of 7129 genes are used. The dataset is publicly available at [21].
Breast cancer dataset [16]
This cDNA microarray dataset contains 76 samples from 2 classes on fiveyear metastasisfree survival: 33 poor prognosis and 43 good prognosis. The expression profiles of 4918 genes are used. The dataset is publicly available at [22]. Other 19 samples with 12 poor prognosis and 7 good prognosis are used as an independent test set as mentioned later.
MLL dataset [17]
This Affymetrix highdensity oligonucleotide array dataset contains 57 samples from 3 classes of leukemia: 20 acute lymphoblastic leukemia (ALL), 17 mixedlineage leukemia (MLL), 20 acute myelogenous leukemia (AML). The expression profiles of 12582 genes are used. The dataset is publicly available at [21]. Note that a test dataset consisting of 15 samples is not used here.
SRBCT dataset [18]
This cDNA microarray dataset contains 63 samples from 4 classes of small round bluecell tumors of childhood (SRBCT): 23 Ewing family of tumors, 20 rhabdomyosarcoma, 12 neuroblastoma, and 8 nonHodgkin lymphoma. The expression profiles of 2308 genes are used. The dataset is publicly available at [23]. Note that a test dataset consisting of 20 SRBCT and 5 nonSRBCT samples is also available, but is not used here.
CNS dataset [15]
This Affymetrix highdensity oligonucleotide array dataset contains 42 samples from 5 different tumors of the central nervous system (CNS): 10 medulloblastomas, 10 malignant gliomas, 10 atypical teratoid/rhabdoid tumors, 8 primitive neuroectodermal tumors, and 4 human cerebella. The expression profiles of 7129 genes are used. The dataset is publicly available at [21].
NCI60 dataset [19]
This cDNA microarray dataset contains 61 samples from 8 classes of human tumor cell lines: 9 breast, 5 CNS, 7 colon, 8 leukemia, 8 melanoma, 9 nonsmall cell lung carcinoma, 6 ovarian, and 9 renal tumors. The expression profiles of 3938 genes are used. The dataset is publicly available at [24].
Preprocessing
For the Prostate cancer, Leukemia, Medulloblastoma, MLL, and CNS datasets, expression values were first thresholded with a floor of 100 (10 for Prostate cancer) and a ceiling of 16000, followed by a base 10 logarithmic transform. Then, each sample was standardized to zero mean and unit variance across genes. For the Colon cancer dataset, after a base 10 logarithmic transform, each sample was standardized. For the Breast cancer dataset, after the filtering of genes following [16], each sample was standardized. For the NCI60 dataset, after filtering genes with missing values, a base 2 logarithmic transform and standardization were applied. For the SRBCT dataset, the expression profiles already preprocessed following [18] were used.
Gene selection methods for comparison
As a baseline gene selection criterion, we employed the S2N ratio [4] for binaryclass problems, and the BW ratio [5] for multiclass problems. Topranked genes with the largest ratios were used for classification. We primarily compared two algorithms for MMCRFE, called uncorrelated MMCRFE and orthogonal MMCRFE (see Methods), with SVMRFE. For the SVM classifier, we used both hardmargin SVM and softmargin SVM with linear kernel. The effect of using support vectors on gene selection may be directly evaluated by hardmargin SVM, i.e. when setting the softmargin parameter C to infinity. The use of softmargin SVM can alleviate the influence of noise and outliers to some extent and avoid overfitting of the data, with the tradeoff between training errors and the margin. In the experiments, we used a wide range of values for the C parameter: C = {0.001, 0.01, 0.1, 1, 10, 100, 1000}. The extension of SVM to more than two classes is not obvious. Hence, several approaches have been proposed for multiclass SVMs, of which we employed oneversusall SVM (OVASVM). Ramaswamy et al. [25] showed the effectiveness of the OVASVM approach for gene selection and classification, and Weston et al. [8] also applied it to gene selection in multiclass problems. In this study, OVASVMbased RFE was performed in the same way as in [8]. For the implementation of SVMRFE, we exploited the Spider library for MATLAB, which is publicly available from [26].
Performance evaluation
We assessed the performance of each gene selection method by repeated random splitting; the samples were partitioned randomly in a class proportional manner into a training set consisting of twothirds of the whole samples and a test set consisting of the heldout onethird of the samples. To avoid selection bias, gene selection was performed using only the training set, and the classification error rate of the learnt classifier was obtained using the test set. This splitting was repeated 100 times. The error rates averaged over the 100 trials and the corresponding standard error rates are reported.
As a baseline classification method, we employed the nearest mean classifier (NMC), which has been found effective for cancer classification [27]. We combined each gene selection method with NMC. Although the nearest neighbor classifier (NNC) was applied as well, NMC consistently showed favorable performance compared with NNC in the repeated random splitting experiments, and thus the results on NMC are reported here. While the performances of the gene selection methods can be compared fair by using the same classifier, SVMRFE is often used as an integrated method of gene selection and classification, and MMCRFE may also perform better when used with the MMC classifier (see Methods). With this view, we further compared the performance between SVMRFE in combination with the SVM classifier and MMCRFE with the MMC classifier. For multiclass datasets, the OVASVM classifier was used.
As suggested by Weston et al. [8], to save computational time of RFE, we removed half of the genes until less than 1000, and then a single gene at a time. In this study, we do not address the problem of finding the optimum number of genes that would yield highest classification accuracy. Instead, the number of genes was varied from 1 to 100, and the performances were compared for each number of genes.
Performance comparison for binaryclass datasets
Tables 1 and 2 show the average error and standard error rates of each combination of classifiers and gene selection criteria for the binaryclass datasets: Colon cancer, Prostate cancer, Leukemia, Medulloblastoma, and Breast cancer. Figures 1 and 2 plot the average error rates as a function of the number of genes from 1 to 100. In the tables and figures, MMCRFE(U), MMCRFE(O), SVMRFE(H) and SVMRFE(S) stand for uncorrelated MMCRFE, orthogonal MMCRFE, hardmargin SVMRFE and softmargin SVMRFE, respectively. For SVMRFE(S), the best result with respect to the C parameter is shown. Our observations from these results are as follows:

NMC+MMCRFE(U,O) versus NMC+SVMRFE(H,S) â€“ Overall, MMCRFE(U,O) shows intermediate performance between SVMRFE(H) and SVMRFE(S) with the best C parameter. MMCRFE(O) is consistently better than MMCRFE(U), and notably MMCRFE(O) performs the best for Leukemia. In most cases, however, the difference is not significant and they are quite competitive.

MMC+MMCRFE(U,O) versus SVM+SVMRFE(H,S) â€“ The performance of MMCRFE(U,O) is improved for Prostate cancer. For the other datasets, the trend is similar to the case of using NMC.

S2N versus MMCRFE(U,O), SVMRFE(H,S) â€“ Both MMCRFE(U,O) and SVMRFE(H,S) improve the performance of NMC over S2N for Prostate cancer, Leukemia and Medulloblastoma. Wessels et al. [27] have reported that NMC with S2N performs the best among various combinations of gene selection methods and classifiers for Colon cancer and Breast cancer. Consistently with their results, S2N performs better than SVMRFE(H) for these datasets. However, a significant improvement is achieved for SVMRFE(S) by setting the C parameter to a small value, e.g. 0.001. Huang and Kecman [28] also reported that the finer tuning of the C parameter can significantly improve the performance of SVMRFE.
Guyon et al. [6] have drawn a conclusion from their result on the Colon cancer dataset that SVMRFE performs better than both S2N and LDARFE. In their experiment, the C parameter was set to 100. However, SVMRFE(S) with C = 100 gives almost the same error rate as SVMRFE(H) for all the binaryclass datasets in our study, and its performance is poorer than that of S2N for Colon cancer, as mentioned previously. There are some reasons that account for this contradiction. First, although Guyon et al. [6] used SVM and weighted voting [3] for classification, we have found that for the Colon cancer dataset, SVM with C = 100 performs significantly worse than NMC when combined with S2N. As can be seen from Table 1, NMC+SVMRFE(H) performs even favorably against SVM+SVMRFE(H). Second, this can be attributed to the selection bias caused by their improper use of crossvalidation [12]; they failed to include the gene selection process in the crossvalidation. Finally, the performance difference between LDARFE and SVMRFE may be due to the difference in the number of genes eliminated at a time.
Guyon et al. [6] also compared the performance between the mean squared errorbased RFE (MSERFE) and SVMRFE, and claimed the superiority of SVMRFE. However, our results suggest that MSERFE might also show better performance in some cases. Indeed, this has been implied by the work of Li and Yang [10], which showed that ridge regressionbased RFE performed better than SVMRFE. It should be noted that MSE is closely related to classical LDA and ridge regression [29, 30]. MMCRFE is still advantageous over LDARFE and MSERFE, because MMCRFE does not need to compute the inverse of a matrix, which makes MMCRFE a computationally efficient and stable method.
As our results indicate, the prediction of clinical outcome is generally more difficult than that of tissue or disease types. The error rates of NMC with S2N for the clinical outcome datasets (Medulloblastoma and Breast Cancer) almost coincide with the results presented in [31], which performed a comparative study on outcome prediction using the same validation strategy as our study. The result for Medulloblastoma shows that the prediction performance can be improved by multivariate gene selection methods such as MMCRFE and SVMRFE. However, it is at best an error rate of above 30% on average, when using twothirds of the samples as a training set.
Performance comparison for multiclass datasets
Tables 3 and 4 show the average error and standard error rates of each combination of classifiers and gene selection criteria for the multiclass datasets: MLL, SRBCT, CNS and NCI60. Figures 3 and 4 plot the average error rates as a function of the number of genes from 1 to 100. The OVASVM approach was used here for SVMRFE. We observe from these results the following:

NMC+MMCRFE(U,O) versus NMC+SVMRFE(H,S) â€“ MMCRFE(U,O) outperforms SVMRFE(H,S) for all the datasets; it shows significantly better performance for a smaller number of genes. MMCRFE(U) appears to be better than MMCRFE(O) for SRBCT, while they are comparable for the other datasets.

MMC+MMCRFE(U,O) versus SVM+SVMRFE(H,S) â€“ The trend is similar to the case of using NMC. Although the performance of NMC+SVMRFE(H,S) is improved by SVM+SVMRFE(H,S) for SRBCT, it is still outperformed by both NMC+MMCRFE(U,O) and MMC+MMCRFE(U,O).

BW versus MMCRFE(U,O), SVMRFE(H,S) â€“ MMCRFE(U,O) shows better performance than BW for three datasets (MLL, SRBCT and NCI60), while performs competitively with BW for CNS. In contrast, SVMRFE(H,S) performs even worse than BW for these datasets, which suggests that OVASVM may not be suitable for selecting a small number of discriminative genes.
Taken together, our extensive comparison has demonstrated that for binaryclass datasets MMCRFE tends to show intermediate performance between hardmargin SVMRFE and SVMRFE with a properly chosen C parameter. Notably, MMCRFE achieves significantly better performance with a smaller number of genes than SVMRFE for multiclass datasets.
The results on hardmargin SVMRFE indicate that the use of support vectors is not necessarily effective for achieving better performance in gene selection. Because the SVMbased approach to gene selection uses the weight vector of the hyperplane constructed by the training samples closest to the decision boundary, the performance can be easily affected by noise and possible outliers. As the results on the binaryclass datasets show, SVMRFE can achieve a significant improvement for some of the datasets by setting the C parameter to a small value. The softmargin parameter plays more roles than just handling noisy data; it is effective for linearly inseparable cases and crucial for avoiding overfitting.
In contrast, MMCRFE uses the discriminant vector obtained by maximizing the average margin, hence less sensitive to noise and outliers. In addition, no parameters need to be tuned. Although MMCRFE may not be so flexible as softmargin SVMRFE, orthogonal MMCRFE shows comparable performance to SVMRFE with the best C parameter for some cases. Another advantage of MMCRFE is that it naturally extends to multiclass cases, while the SVMbased approach typically treats them by decomposing the multiclass problems into many binaryclass ones, e.g. oneversusone and oneversusall strategies. Therefore, MMCRFE is in particular effective for gene selection in multiclass problems, which has also been validated by the performance on the multiclass datasets.
Comparison of selected genes
It is clearly of interest to compare the selected genes of MMCRFE with those of S2N and SVMRFE. To this end, we conducted additional experiments using independent test sets. The results were obtained for the Prostate cancer, Leukemia, and Breast cancer datasets. Note that the test set for Prostate cancer is from [32], which is available at [33]. It contains 25 normal tissue samples and 9 prostate tumor samples. Gene selection was performed using the whole samples in the previous experiment, and the classification error rate of the learnt classifier was obtained using the independent test set. NNC and NMC were used here for classification, and the number of genes was varied from 1 to 100.
For each dataset, the minimum number of misclassifications as well as the number of genes used are shown in Table 5. Both uncorrelated MMCRFE and orthogonal MMCRFE achieve zero misclassifications using a small number of genes for Prostate cancer and Leukemia, although S2N and SVMRFE also perform comparably well. While S2N and MMCRFE misclassify three or four test samples for Breast cancer, SVMRFE yields fewer misclassifications with NMC by using C = 0.001.
Tables 6, 7, 8 list the 10 topranked genes of orthogonal MMCRFE and the corresponding ranks by S2N and SVMRFE. Note that the lists of uncorrelated MMCRFE are similar to those of orthogonal MMCRFE, and hence omitted. For Prostate cancer, 7 genes are included in the list of 16 genes identified by Singh et al. [14] (Table 6). Of note, HPN (X07732) is ranked the first by all the three gene selection methods. It is known that hepsin, a cell surface serine protease, is overexpressed in prostate cancer and has been identified as a potential prostatecancer biomarker [32, 34â€“36]. HPN and CFD (M84526) are the two genes that are selected by uncorrelated MMCRFE and achieve perfect classification with NMC. We can see that some of these genes are also highly ranked by S2N and SVMRFE. Despite that there are differences in the preprocessing steps and gene selection criteria used, half the genes are included in the lists of the original studies for Leukemia and Breast cancer (Tables 7 and 8); the number of genes identified and used for prediction was 50 for Leukemia [3] and 70 for Breast cancer [16], respectively. It appears that the topranked genes of orthogonal MMCRFE show a larger overlap with those of SVMRFE than with those of S2N. Indeed, almost all the listed genes belong to the 50 topranked genes of SVMRFE. On the other hand, some of them are assigned small ratio values by S2N. This may be due to the difference in gene selection approaches; MMCRFE and SVMRFE are multivariate, whereas S2N is univariate. It is interesting to note that the first rank gene of SVMRFE for Breast cancer is PRAME (NM_006115), which in combination with TSPYL5 (AL080059) yields only one misclassification with NMC. The rank of PRAME by orthogonal MMCRFE and S2N is 33 and 107, respectively. Overall, these results show that MMCRFE can identify a small subset of discriminative genes that is quite consistent with previous studies.
Conclusion
In this paper, we have proposed a recursive gene selection method based on the MMC, and presented efficient and stable algorithms for MMCRFE. The performance of MMCRFE was extensively compared with that of SVMRFE using nine cancer microarray datasets, including four multiclass datasets. We further compared the topranked genes selected by MMCRFE with those of other gene selection methods, showing the validity of MMCRFE.
The results suggest that MMCRFE is less sensitive to noise and outliers due to the use of average margin, while the performance of SVMRFE can be easily affected by them when applied to noisy, small sample size microarray data. Another advantage of MMCRFE over SVMRFE is that MMCRFE naturally extends to multiclass cases. Furthermore, MMCRFE does not require the computation of the matrix inversion unlike LDARFE and MSERFE, and involves no parameters to be tuned.
This study has shown the effectiveness of the MMC for gene selection using microarray data. Our proposed algorithms can also be applied to proteomics and metabolomics datasets, and may be useful for biomarker discovery from such noisy data.
Methods
Maximum margin criterion
Linear discriminant analysis (LDA) aims to find a set of projection vectors which maximize the betweenclass scatter and simultaneously minimize the withinclass scatter, thereby achieving maximum discrimination [37].
The betweenclass scatter matrix S_{ b }and the withinclass scatter matrix S_{ w }are defined as
where c is the number of classes, m_{ i }and p_{ i }are the mean and a priori probability of class i, m is the total mean, and S_{ i }is the covariance matrix of class i. Then, classical LDA finds the projection vectors W by maximizing the Fisher criterion
J (W) = trace ((W^{T}S_{ w }W)^{1} (W^{T}S_{ b }W)). Â Â Â (1)
By solving a generalized eigenvalue problem, the projection vectors W can be found as the eigenvectors of {S}_{w}^{\xe2\u02c6\u20191}S_{ b }corresponding to the largest eigenvalues. When the sample size is smaller than the dimensionality of samples, however, S_{ w }becomes singular and we cannot compute {S}_{w}^{\xe2\u02c6\u20191}S_{ b }, which is a major drawback of classical LDA.
To overcome the singularity problem, several methods have been proposed e.g. in the field of computer vision, where the number of samples is usually much smaller than the dimensionality. A simple approach is to replace {S}_{w}^{\xe2\u02c6\u20191} with the pseudoinverse matrix {S}_{w}^{+}. Another approach is to add some constant values to the diagonal elements of S_{ w }as S_{ w }+ Î¼ I, where Î¼ > 0 and I is the identity matrix. However, each of these methods has its own drawbacks and does not scale well to highdimensional data (see [11] for more details). Recently, Li et al. [11] proposed to use the maximum margin criterion (MMC) instead of (1) to find the projection vectors. The MMC is defined as
J (W) = trace (W^{T} (S_{ b } S_{ w }) W)). Â Â Â (2)
The projection vectors W= (w_{1},..., w_{ d }) which maximize (2) under the constraint that {w}_{k}^{T}w_{ k }= 1, k = 1,..., d, can be found as the eigenvectors of S_{ b } S_{ w }corresponding to the largest eigenvalues. The advantage of using the MMC is that we need not compute the inverse of S_{ w }, hence the singularity problem can be easily avoided.
It is known that classical LDA can be related to SVM. Shashua [38] has shown that, in binaryclass cases, the orientation and location of the hyperplane obtained by SVM is equivalent to the discriminant vector obtained by classical LDA using the samples on the margin. In other words, SVM can be viewed as sparsified LDA. Thus, noting that the MMC is different from classical LDA only in its constraint [11], the major difference between SVM and the MMC consists in that the hyperplane of SVM is constructed only by the training samples closest to the decision boundary, while the discriminant vector of the MMC is constructed so that the average margin computed by all training samples is maximized. They also lead to different problems to solve: a quadratic programming problem for the standard L_{2} SVM and an eigenvalue problem for the MMC. Note that for L_{1} SVM, it can be reduced to a linear programming problem (see [9] and references therein).
MMCRFE algorithms for gene selection
The idea of recursive feature elimination (RFE) [6] is to recursively remove genes using the absolute weights of the discriminant vector or hyperplane, which reflect the significance of the genes for classification. The process starts by training the classifier using all genes. Then, the genes are ranked according to the absolute weights, and those genes with the smallest absolute weights are removed. The classifier is retrained with the remaining genes. This process is repeated until the maximum classification accuracy is obtained or the number of genes reaches a predetermined value. The RFE approach has recently been shown to be effective not only with SVM but also with penalized logistic regression [39] and ridge regression [10].
Here, we propose a recursive gene selection method based on the MMC. The MMC is computationally more efficient and stable than classical LDA, yet it does not scale well to highdimensional data. When we consider using RFE with the MMC, it is computationally intensive to perform the eigenvalue decomposition (EVD) of the matrix of the gene size in a recursive manner. To overcome the problem of high dimensionality, we first remove the null space of the total scatter matrix via singular value decomposition (SVD) [40], thereby reduce the dimensionality of the data to n  1, where n is the number of samples, and then maximize the MMC in the reduced space. Let X denote the gene expression matrix of size p Ã— n, where p is the number of genes. Then, the total scatter matrix S_{ t }can be expressed as
where
and e= (1,1,..., 1)^{T} is an ndimensional vector. Let us assume that p > n and perform the reduced SVD of \stackrel{\xcb\u0153}{X} as
where \stackrel{\xcb\u0153}{\mathrm{\xce\u203a}} = diag (Î»_{1},..., Î»_{ n }) with decreasing nonnegative values, and \stackrel{\xcb\u0153}{U} and \stackrel{\xcb\u0153}{V} are p Ã— n and n Ã— n orthonormal matrices. Since the rank of S_{ t }is n  1, i.e. Î»_{ n }= 0, we can rewrite (3) as
= U Î› V^{T},
where Î› = diag (Î»_{1},..., Î»_{n  1}), and U and V are p Ã— (n  1) and n Ã— (n  1) matrices consisting of the corresponding (n  1) vectors. Thus, we can reduce the dimension by projecting X onto the (n  l)dimensional space as
Z= Î›^{1} U^{T}X. Â Â Â (4)
Then, we may maximize the MMC on Z, which is a (n  1) Ã— n matrix. Here, we require W to be orthogonal, i.e. W^{T}W= I, in the reduced space. Once the discriminant vectors W of size (n  1) Ã— d is obtained, they are projected back onto the original pdimensional space by
= U Î›^{1} W, Â Â Â (5)
where \stackrel{\xcb\u0153}{W} is of size p Ã— d. Finally, gene selection can be performed using \stackrel{\xcb\u0153}{W}. When using (4), we can show that the number of the discriminant vectors that correspond to the positive eigenvalues is at most c  1. Because the eigenvalues reflect the discrimination ability, we use the (c  1) discriminant vectors corresponding to the positive eigenvalues, i.e. d is set to c  1, and discard those corresponding to the negative eigenvalues.
Li et al. [11] proposed another efficient method to compute the projection vectors of the MMC. It is interesting to note that the MMC is related to uncorrelated LDA (ULDA), and we can find that the Li's method is the same as the ULDA algorithm proposed by Ye [41]. It can be shown that \stackrel{\xcb\u0153}{W} in (5) maximizes the MMC on X under the constraint that \stackrel{\xcb\u0153}{W}^{T}S_{ t }\stackrel{\xcb\u0153}{W} = I, and our method turns out to be equivalent to the ULDA algorithm. Hence, we call the algorithm based on (4) uncorrelated MMCRFE.
This study also explores the following projection instead of (4):
Z= U^{T}X. Â Â Â (6)
After obtaining the discriminant vectors W by maximizing the MMC on Z, they are projected back onto the original pdimensional space by
= U W. Â Â (7)
Note that no discriminant information is lost in the case of (6) [42]. It can be shown that \stackrel{\xcb\u0153}{W} in (7) maximizes the MMC on X under the constraint that \stackrel{\xcb\u0153}{W}^{T}\stackrel{\xcb\u0153}{W} = I. We call the algorithm based on (6) orthogonal MMCRFE. We see that the difference between (4) and (6) results in the different constraints of the MMC on X.
The uncorrelated MMCRFE and orthogonal MMCRFE algorithms are summarized in Figures 5 and 6, respectively. They are different in step 3 and step 6. The main computation of both algorithms consists of the SVD of a p Ã— n matrix at step 2 and the EVD of a (n  1) Ã— (n  1) matrix at step 5. Thus, the algorithms are feasible in the case of high dimensionality and small sample size, i.e. large p and small n. As is shown, the MMCRFE algorithms can naturally treat multiclass cases, in which the weight of gene j can be defined as the sum of the absolute weights of c  1 discriminant vectors in \stackrel{\xcb\u0153}{W}, i.e. {\displaystyle {\xe2\u02c6\u2018}_{k=1}^{c\xe2\u02c6\u20191}\left{\stackrel{\xcb\u0153}{w}}_{jk}\right}. The maximum of the absolute weights, i.e. max_{k = 1,..., c  1}\stackrel{\xcb\u0153}{w}_{ jk }, may also be useful. Note that the uncorrelated MMCRFE algorithm switches to orthogonal MMCRFE at q = n  1, where q is the number of remaining genes during elimination. Hence, our algorithm and the ULDA algorithm may select different genes when q â‰¤ n  1.
MMC classifier
The MMC classifier performs nearest mean classification in the projected space, i.e. the class label y of a test sample x is predicted as
where m_{ i }is the mean of class i. Since we perform classification using at most 100 genes in the experiments, the discriminant vectors W were computed by directly maximizing the MMC under the orthogonality constraint.
S2N ratio and BW ratio
For each gene j, the S2N ratio [4] is defined as
where {\mathrm{\xce\xbc}}_{j}^{(1)}, {\mathrm{\xce\xbc}}_{j}^{(2)} and {\mathrm{\xcf\u0192}}_{j}^{(1)}, {\mathrm{\xcf\u0192}}_{j}^{(2)} denote the means and standard deviations of two classes, respectively. Topranked genes are those with the largest values of S2N(j).
The BW ratio [5] can be defined as
where \stackrel{\xc2\xaf}{x}{.}_{j}^{(k)} and \stackrel{\xc2\xaf}{x}{.}_{j} respectively denote the average expression level of gene j for class k and the overall average expression level of gene j across all samples, y_{ i }denotes the class of sample i, and I (Â·) is the indicator function. Topranked genes with the largest values of BW(j) are used for classification.
References
Stolovitzky G: Gene selection in microarray data: the elephant, the blind men and our algorithms. Curr Opin Struct Biol 2003, 13: 370â€“376. 10.1016/S0959440X(03)000782
CuperlovicCulf M, Belacel N, Ouellette RJ: Determination of tumour marker genes from gene expression data. Drug Discov Today 2005, 10: 429â€“437. 10.1016/S13596446(05)033933
Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, Mesirov JP, Coller H, Loh ML, Downing JR, Caligiuri MA, Bloomfield CD, Lander ES: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 1999, 286: 531â€“537. 10.1126/science.286.5439.531
Furey TS, Cristianini N, Duffy N, Bednarski DW, Schummer M, Haussler D: Support vector machine classification and validation of cancer tissue samples using microarray expression data. Bioinformatics 2000, 16: 906â€“914. 10.1093/bioinformatics/16.10.906
Dudoit S, Fridlyand J, Speed TP: Comparison of discrimination methods for the classification of tumors using gene expression data. J Amer Statist Assoc 2002, 97: 77â€“87. 10.1198/016214502753479248
Guyon I, Weston J, Barnhill S, Vapnik V: Gene selection for cancer classification using support vector machines. Mach Learn 2002, 46: 389â€“422. 10.1023/A:1012487302797
Rakotomamonjy A: Variable selection using SVMbased criteria. J Mach Learn Res 2003, 3: 1357â€“1370. [http://jmlr.csail.mit.edu/papers/v3/rakotomamonjy03a.html] 10.1162/153244303322753706
Weston J, Elisseeff A, SchÃ¶lkopf B, Tipping M: Use of the zeronorm with linear models and kernel methods. J Mach Learn Res 2003, 3: 1439â€“1461. [http://jmlr.csail.mit.edu/papers/v3/weston03a.html] 10.1162/153244303322753751
Zhang HH, Ahn J, Lin X, Park C: Gene selection using support vector machines with nonconvex penalty. Bioinformatics 2006, 22: 88â€“95. 10.1093/bioinformatics/bti736
Li F, Yang Y: Analysis of recursive gene selection approaches from microarray data. Bioinformatics 2005, 21: 3741â€“3747. 10.1093/bioinformatics/bti618
Li H, Jiang T, Zhang K: Efficient and robust feature extraction by maximum margin criterion. IEEE Trans Neural Netw 2006, 17: 157â€“165. 10.1109/TNN.2005.860852
Ambroise C, McLachlan GJ: Selection bias in gene extraction on the basis of microarray geneexpression data. Proc Natl Acad Sci USA 2002, 99: 6562â€“6566. 10.1073/pnas.102102699
Alon U, Barkai N, Notterman DA, Gish K, Ybarra S, Mack D, Levine AJ: Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc Natl Acad Sci USA 1999, 96: 6745â€“6750. 10.1073/pnas.96.12.6745
Singh D, Febbo PG, Ross K, Jackson DG, Manola J, Ladd C, Tamayo P, Renshaw AA, D'Amico AV, Richie JP, Lander ES, Loda M, Kantoff PW, Golub TR, Sellers WR: Gene expression correlates of clinical prostate cancer behavior. Cancer Cell 2002, 1: 203â€“209. 10.1016/S15356108(02)000302
Pomeroy SL, Tamayo P, Gaasenbeek M, Sturla LM, Angelo M, McLaughlin ME, Kim JYH, Goumnerova LC, Black PM, Lau C, Allen JC, Zagzag D, Olson JM, Curran T, Wetmore C, Biegel JA, Poggio T, Mukherjee S, Rifkin R, Califano A, Stolovitzky G, Louis DN, Mesirov JP, Lander ES, Golub TR: Prediction of central nervous system embryonal tumour outcome based on gene expression. Nature 2002, 415: 436â€“442. 10.1038/415436a
van't Veer LJ, Dai H, van de Vijver MJ, He YD, Hart AAM, Mao M, Peterse HL, van der Kooy K, Marton MJ, Witteveen AT, Schreiber GJ, Kerkhoven RM, Roberts C, Linsley PS, Bernards R, Friend SH: Gene expression profiling predicts clinical outcome of breast cancer. Nature 2002, 415: 530â€“536. 10.1038/415530a
Armstrong SA, Staunton JE, Silverman LB, Pieters R, den Boer ML, Minden MD, Sallan SE, Lander ES, Golub TR, Korsmeyer SJ: MLL translocations specify a distinct gene expression profile that distinguishes a unique leukemia. Nat Genet 2002, 30: 41â€“47. 10.1038/ng765
Khan J, Wei JS, RingnÃ©r M, Saal LH, Ladanyi M, Westermann F, Berthold F, Schwab M, Antonescu CR, Peterson C, Meltzer PS: Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks. Nat Med 2001, 7: 673â€“679. 10.1038/89044
Ross DT, Scherf U, Eisen MB, Perou CM, Rees C, Spellman P, Iyer V, Jeffrey SS, Van de Rijn M, Waltham M, Pergamenschikov A, Lee JCF, Lashkari D, Shalon D, Myers TG, Weinstein JN, Botstein D, Brown PO: Systematic variation in gene expression patterns in human cancer cell lines. Nat Genet 2000, 24: 227â€“235. 10.1038/73432
Colon cancer dataset[http://microarray.princeton.edu/oncology/affydata/index.html]
Cancer Program Data Sets[http://www.broad.mit.edu/cgibin/cancer/datasets.cgi]
Rosetta Inpharmatics[http://www.rii.com/publications/2002/vantveer.html]
Microarray Project[http://research.nhgri.nih.gov/microarray/Supplement/]
Stanford NCI60 Cancer Microarray Project[http://genomewww.stanford.edu/nci60/]
Ramaswamy S, Tamayo P, Rifkin R, Mukherjee S, Yeang CH, Angelo M, Ladd C, Reich M, Latulippe E, Mesirov JP, Poggio T, Gerald W, Loda M, Lander ES, Golub TR: Multiclass cancer diagnosis using tumor gene expression signatures. Proc Natl Acad Sci USA 2001, 98: 15149â€“15154. 10.1073/pnas.211566398
The Spider Library for MATLAB[http://www.kyb.tuebingen.mpg.de/bs/people/spider/]
Wessels LFA, Reinders MJT, Hart AAM, Veenman CJ, Dai H, He YD, van't Veer LJ: A protocol for building and evaluating predictors of disease state based on microarray data. Bioinformatics 2005, 21: 3755â€“3762. 10.1093/bioinformatics/bti429
Huang TM, Kecman V: Gene extraction for cancer diagnosis by support vector machines â€“ An improvement. Artif Intell Med 2005, 35: 185â€“194. 10.1016/j.artmed.2005.01.006
Duda RO, Hart PE, Stork DG: Pattern Classification. 2nd edition. New York: Wiley; 2001.
Xu J, Zhang X, Li Y: Kernel MSE algorithm: A unified framework for KFD, LSSVM and KRR. In Proceedings of the International Joint Conference on Neural Networks: 15â€“19 July 2001. Washington, DC, IEEE; 2001:1486â€“1491.
Michiels S, Koscielny S, Hill C: Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet 2005, 365: 488â€“492. 10.1016/S01406736(05)178660
Welsh JB, Sapinoso LM, Su AI, Kern SG, WangRodriguez J, Moskaluk CA, Frierson HF Jr, Hampton GM: Analysis of gene expression identifies candidate markers and pharmacological targets in prostate cancer. Cancer Res 2001, 61: 5974â€“5978.
Prostate cancer data from GNF[http://www.gnf.org/cancer/prostate]
Dhanasekaran SM, Barrette TR, Ghosh D, Shah R, Varambally S, Kurachi K, Pienta KJ, Rubin MA, Chinnaiyan AM: Delineation of prognostic biomarkers in prostate cancer. Nature 2001, 412: 822â€“826. 10.1038/35090585
Luo J, Duggan DJ, Chen Y, Sauvageot J, Ewing CM, Bittner ML, Trent JM, Isaacs WB: Human prostate cancer and benign prostatic hyperplasia: Molecular dissection by gene expression profiling. Cancer Res 2001, 61: 4683â€“4688.
Magee JA, Araki T, Patil S, Ehrig T, True L, Humphrey PA, Catalona WJ, Watson MA, Milbrandt J: Expression profiling reveals hepsin overexpression in prostate cancer. Cancer Res 2001, 61: 5692â€“5696.
Fukunaga K: Introduction to Statistical Pattern Recognition. 2nd edition. Boston, MA: Academic Press; 1990.
Shashua A: On the relationship between the support vector machine for classification and sparsified Fisher's linear discriminant. Neural Processing Lett 1999, 9: 129â€“139. 10.1023/A:1018677409366
Zhu J, Hastie T: Classification of gene microarrays by penalized logistic regression. Biostatistics 2004, 5: 427â€“443. 10.1093/biostatistics/kxg046
Golub GH, Van Loan CF: Matrix Computations. 3rd edition. Baltimore, MD: Johns Hopkins University Press; 1996.
Ye J: Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems. J Mach Learn Res 2005, 6: 483â€“502. [http://jmlr.csail.mit.edu/papers/v6/ye05a.html]
Yang J, Yang JY: Why can LDA be performed in PCA transformed space? Pattern Recognition 2003, 36: 563â€“566. [http://dx.doi.org/10.1016/S0031â€“3203(02)00048â€“1] 10.1016/S00313203(02)002625
Acknowledgements
This work was supported by KAKENHI (GrantinAid for Scientific Research) on Priority Areas "Comparative Genomics" from the Ministry of Education, Culture, Sports, Science and Technology of Japan.
Author information
Authors and Affiliations
Corresponding author
Additional information
Authors' contributions
SN designed the experiments, carried out all the analysis, and drafted the manuscript. SK supervised the project. All authors read and approved the final manuscript.
Authorsâ€™ original submitted files for images
Below are the links to the authorsâ€™ original submitted files for images.
Rights and permissions
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Cite this article
Niijima, S., Kuhara, S. Recursive gene selection based on maximum margin criterion: a comparison with SVMRFE. BMC Bioinformatics 7, 543 (2006). https://doi.org/10.1186/147121057543
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/147121057543