- Methodology Article
- Open Access
Statistical analysis of a Bayesian classifier based on the expression of miRNAs
- Leonardo Ricci^{1}Email author,
- Valerio Del Vescovo^{2},
- Chiara Cantaloni^{3},
- Margherita Grasso^{2},
- Mattia Barbareschi^{3} and
- Michela Alessandra Denti^{2}
- Received: 10 April 2015
- Accepted: 24 August 2015
- Published: 4 September 2015
Abstract
Background
During the last decade, many scientific works have concerned the possible use of miRNA levels as diagnostic and prognostic tools for different kinds of cancer. The development of reliable classifiers requires tackling several crucial aspects, some of which have been widely overlooked in the scientific literature: the distribution of the measured miRNA expressions and the statistical uncertainty that affects the parameters that characterize a classifier. In this paper, these topics are analysed in detail by discussing a model problem, i.e. the development of a Bayesian classifier that, on the basis of the expression of miR-205, miR-21 and snRNA U6, discriminates samples into two classes of pulmonary tumors: adenocarcinomas and squamous cell carcinomas.
Results
We proved that the variance of miRNA expression triplicates is well described by a normal distribution and that triplicate averages also follow normal distributions. We provide a method to enhance a classifiers’ performance by exploiting the correlations between the class-discriminating miRNA and the expression of an additional normalized miRNA.
Conclusions
By exploiting the normal behavior of triplicate variances and averages, invalid samples (outliers) can be identified by checking their variability via chi-square test or their displacement by the respective population mean via Student’s t-test. Finally, the normal behavior allows to optimally set the Bayesian classifier and to determine its performance and the related uncertainty.
Keywords
- microRNA
- Bayesian classifiers
- Lung cancer
- qRT-PCR gene expression measurement
Background
MicroRNAs (miRNAs or miRs) are small non-coding single-stranded RNAs, 19–25 nucleotides in length, acting as negative regulators of gene expression at the post-transcriptional level. More than 1000 miRNAs are transcribed from miRNA genes in the human genome. A single miRNA is able to modulate hundreds of downstream genes by recognizing complementary sequences in the 3^{′} untranslated regions (UTRs) of their target messenger RNAs. It has been estimated that in humans about 30 % of messenger RNAs are under miRNA regulation. The biological functions of miRNAs are diverse and include several key cellular processes, such as differentiation, proliferation, cellular development, cell death and metabolism.
In the last decade, evidences have accumulated to indicate that miRNAs play a role in the onset and progression of several human cancers [1]. The transcription or processing of some miRNAs is altered in neoplastic tissues, with respect to their normal counterparts. miRNAs whose levels increase in tumors are referred to as oncogenic miRNAs (onco-miRs), sometimes even if there is no evidence for their causative role in tumorigenesis. On the other hand, miRNAs down-regulated in cancer are considered tumor suppressors.
In parallel to these studies, the effectiveness of miRNAs as markers for tracing the tissue of origin of cancers of unknown primary origin was demonstrated by several authors, and the utility of miRNAs levels as diagnostic and prognostic markers became clear (reviewed by [2]). The main advantage of the use of miRNAs as markers resides in the ease of their detection and in their extreme specificity. miRNAs are stable molecules well preserved in formalin fixed, paraffin embedded tissues (FFPE) as well as in fresh snap-frozen specimens, unlike larger RNA molecules as messenger RNAs [3]. The finding that miRNAs have an exceptional stability in several tissues suggested that these tiny molecules were also preserved, detectable and quantifiable in plasma and in other biofluids, such as urine, saliva, cerebro-spinal fluid and amniotic fluid [4]. Circulating miRNAs are attracting attention as markers not only for cancer but also for neurodegenerative diseases (reviewed by [5]) as they have some important features: non-invasivity, specificity, early detection, sensitivity and ease of translatability from model systems to humans.
Methods based on next generation sequencing (NGS), microarray and quantitative reverse-transcription polymerase chain reaction (qRT-PCR) are currently used for miRNA profiling and for the identification of miRNAs differently expressed in tumor samples and in matched, healthy tissue. The majority of miRNA profiling studies have been so far carried out by using microarrays. These studies have provided signatures consisting in few to several (5–30) distinct miRNAs [6]. However, the large amount of data obtained by microarray and NGS profiling needs to be transposed into clinical trials by developing an easily performed, cost-effective and serviceable assay that can analyze the cancer-specific miRNAs for cancer diagnosis and prognosis. Such an analysis has been so far relying on qRT-PCR assays, performed by measuring the levels of a restricted number of miRNAs (see review by [7]).
The realization of classifiers based on the expression of miRNAs is widely discussed in the scientific literature. Within the context of lung cancers (see, for example, [4, 8–12]) the work by [13] describes a classifier that distinguishes squamous from nonsquamous non-small-cell lung carcinomas, by using miR-205 as a specific marker and miR-21, snRNA (small nuclear RNA) U6 as normalizers. The approach followed is essentially machine learning: the classifier relies on a sample score and a threshold. A more elaborated support vector machine, which uses the combination of 5 miRNAs for lung squamous cell carcinoma diagnosis, is described in [14]. A receiver operating characteristic curve analysis to evaluate the possibility of diagnosing the histologic subtype of pulmonary neuroendocrine tumors via altered expression of miR-21, miR-155, let-7a is discussed in [15] (a similar statistical approach is described in [16]).
These classifiers are generally declared to be efficient. For example, [13] report a sensitivity of 96 % and a specificity of 90 %. However, at least two aspects are widely overlooked in the scientific literature: first, the distribution of the measured miRNA expressions; second, the statistical uncertainty that unavoidably affects the parameters that characterize a classifier and its performance. Both aspects are crucial in order to assess the reproducibility, and thus the reliability, of a classifier. The goal of the present paper is to close these gaps. Our analysis concerns a Bayesian classifier based on the expression of a single class-discriminating miRNA, with additional miRNAs that are used either as normalizers or as performance-enhancer via noise-reduction.
In the present paper the following issues are discussed: normal distribution of the triplicate variance and identification of outliers; improvement of accuracy via normalizers; class-discriminating measures and their distributions; identification of “bias” outliers; assessment of a classifier’s performance; finally, improvement of a classifier’s performance by exploiting correlations.
As a prototypical case we discuss throughout the paper the development of a classifier that assigns samples either to adenocarcinomas (ADC) or to squamous cell carcinomas (SQC). The two classes ADC, SQC are henceforth referred to as the target and the versus class, respectively. The miRNAs used are miR-205, miR-21 and snRNA U6.
Methods
Distribution of triplicates: normally-distributed variance and outlier identification
Given a sample stemming from a patient, a set of miRNAs is measured in triplicate by using qRT-PCR. For each miRNA, the sample mean x of the corresponding triplicate and the related sample standard deviation s are calculated. To provide an a priori knowledge on the samples, each one was classified via immunohistochemical analysis and gene profiling into one of different categories of lung tumors. We use data based on lung carcinoma biopsies retrieved from the archives of the Unit of Surgical Pathology of the S. Chiara Hospital in Trento, Italy. The research project had been approved by the Ethical Committee of the Trentino Public Health System (Azienda Provinciale per i Servizi Sanitari). Most of the data analyzed here were previously published by our research group [17]. All datasets are available as Supplementary material in the Additional file 1.
Statistics of the standard deviation for the sets of triplicates of miR-205, miR-21 and snRNA U6, as well as for the snRNA U6 set devoid of outliers
miRNA | Set size | σ | Fit’s p-value | σ _{max} | 〈s ^{2}〉^{1/2} | [ 〈s ^{4}〉−〈s ^{2}〉^{2}]^{1/4} |
---|---|---|---|---|---|---|
miR-205 | 37 | 0.25 | 0.60 | 0.61 | 0.26 | 0.25 |
miR-21 | 39 | 0.19 | 0.99 | 0.46 | 0.20 | 0.21 |
snRNA U6 | 39 | 0.19 | 0.10 | 0.48 | 0.26 | 0.30 |
snRNA U6 | 35 | 0.17 | 0.13 | - | 0.20 | 0.21 |
without outliers |
As a result, an outlier can be identified by checking its variability via the chi-square test: we assume a triplicate of a given miRNA to be an outlier if its sample standard deviation s exceeds the critical value σ _{max} corresponding to the significance level α=0.05. The critical value σ _{max} is given by [−2 logα]^{1/2} σ≃2.448 σ, where σ is the population standard deviation assessed for that miRNA. The σ _{max} values are reported in Table 1. The outlier definition used here implies that the significance level α corresponds to the rate of statistical false alarms (type I errors), i.e. the rate of valid triplicates that are falsely deemed to be outliers.^{1}
According to this procedure, the triplicate sets of both miR-205 and miR-21 contained no outliers, whereas the snRNA U6 contained 4 outliers. These samples were excluded from the following analysis. Table 1 also shows the statistics of the standard deviation for the snRNA U6 data set devoid of outliers.
We note that, for snRNA U6, the two values of σ are very similar. This fact reflects, as mentioned above, the robustness to outliers of the K-S approach. In addition, for all data sets devoid of outliers, the population standard deviation σ is approximately equal to both the root-mean-square sample standard deviation 〈s ^{2}〉^{1/2} and the fourth root of the variance of variances [〈s ^{4}〉−〈s ^{2}〉^{2}]^{1/4}. Because ν=2, this behavior provides further evidence to the null hypothesis that samples are drawn from the same normal distribution.
For each sample, i.e. patient, of the set devoid of outliers, we henceforth use the following notation for the sample mean x of the available triplicates: x _{U6} for the snRNA U6 triplicate, x _{21} for the miR-21 triplicate, and x _{205} for the miR-205 triplicate. In addition, x _{205}, x _{21}, x _{U6} will be referred to as measures.
Distribution of triplicates: accuracy and normalization
A main issue to cope with towards the development of a reliable classifier is accuracy. The question is whether the values of the sample means of the triplicates are constant over different experimental sessions – i.e. measurements taken at different times and/or with different set-ups – or, rather, have to be normalized in order to remove experimental bias.
Statistics of the measures x _{21}, x _{U6}, x _{21}−x _{U6} obtained on data stemming from two different experimental sessions
Session | Measure | Class | Set size | \(\overline {X}\) | S |
---|---|---|---|---|---|
I | x _{21} | target | 21 | 18.4(2) | 1.0(2) |
versus | 18 | 19.2(6) | 2.6(5) | ||
x _{U6} | target | 19 | 25.0(3) | 1.4(2) | |
versus | 16 | 25.8(5) | 1.9(3) | ||
x _{21}−x _{U6} | target | 19 | –6.5(3) | 1.4(2) | |
versus | 16 | –6.6(6) | 2.4(4) | ||
II | x _{21} | target | 19 | 21.4(3) | 1.1(2) |
versus | 17 | 21.9(6) | 2.3(4) | ||
x _{U6} | target | 20 | 27.4(3) | 1.2(2) | |
versus | 16 | 28.4(6) | 2.6(5) | ||
x _{21}−x _{U6} | target | 17 | –6.1(3) | 1.1(2) | |
versus | 16 | –6.8(6) | 2.5(5) |
The necessity to improve accuracy by suitably normalizing an oncomir has been extensively discussed in the scientific literature (see for example [18]). Thus, in accordance with many previous works, we use snRNA U6 to normalize x _{21} and x _{205}. The following notation is henceforth used: Δ x _{205}≡x _{205}−x _{U6}; Δ x _{21}≡x _{21}−x _{U6}.
MiRNA statistics
In this section, the statistics of Δ x _{205}, Δ x _{21}, snRNA U6 is discussed.
Measure | Class | Set size | \(\overline {X}\) | S | TN | t-test |
---|---|---|---|---|---|---|
Δ x _{205} | target | 18 | 1.7(5) | 2.2(4) | 0.71 | 2·10^{−8} |
versus | 15 | –4.6(6) | 2.4(5) | 0.26 | ||
Δ x _{21} | target | 19 | –6.5(3) | 1.4(2) | 0.53 | 0.90 |
versus | 16 | –6.6(6) | 2.4(4) | 0.94 | ||
x _{U6} | target | 19 | 25.0(3) | 1.4(2) | 0.40 | 0.14 |
versus | 16 | 25.8(5) | 1.9(3) | 0.25 | ||
y _{ DV } | target | 18 | 4.9(5) | 2.0(3) | 0.96 | 5.5·10^{−11} |
versus | 15 | –1.3(4) | 1.7(3) | 0.98 | ||
y _{ opt } | target | 18 | 6.9(5) | 1.9(3) | 0.78 | 2.6·10^{−11} |
versus | 15 | 0.7(4) | 1.6(3) | 0.24 |
By means of the Shapiro-Wilk test of normality, each histogram was shown to be consistent with a normal parent population. Consequently, we assume that, for each of the two classes, the measures Δ x _{205}, Δ x _{21}, x _{U6} are normally distributed with a mean and a standard deviation that are respectively estimated by the sample mean \(\overline {X}\) and the sample standard deviation S of the x values (triplicates). The results are reported in Table 3. The proof that the measures of interest are compatible with normal distributions makes up a crucial step towards the optimization of the Bayesian classifier and the determination of its performance, inclusively of the related uncertainty (see below).
With regard to Δ x _{205}, the histograms of the target class ADC and the versus class SQC are well-separated: Student’s t-test provides p<10^{−7} (see Table 3). Conversely, both for Δ x _{21} and x _{U6}, the overlapping of the histograms of the two classes is confirmed by Student’s t-test, which provides p=0.90 and p=0.14, respectively. Therefore, only the measure Δ x _{205} is a good candidate to classify samples into ADC or SQC.
A Bayesian classifier
was used and a threshold χ=2.5 was set.
The discriminator approach described in Eq. (1) requires tackling three main issues: finding a suitable linear combination y; finding a suitable value for χ; analyzing the performance of the classifier.
The uncertainties on the three coefficients η, β, γ, and thus on the threshold χ, are computed by means of standard error propagation. We remind that μ _{ T }, σ _{ T }, μ _{ V }, σ _{ V } are evaluated as sample means and sample standard deviations, and are therefore uncertainty-affected: for example, the errors on μ _{ T }, σ _{ T } are \({\sigma _{T}/\sqrt {N_{T}}}\), \({\sigma _{T}/\sqrt {2(N_{T}-1)}}\), respectively, where N _{ T } is the number of triplicates belonging to the target class.
Measure | χ _{10:90} | χ | χ _{90:10} |
---|---|---|---|
Δ x _{205} | –3.2(7) | –1.3(4) | 0.6(8) |
Δ x _{21} | –10(1) | –8.2(6) | –6.4(6) |
−x _{U6} | –29(1) | –26.1(5) | –24(1) |
y _{ DV } | 0.5(5) | 1.7(4) | 2.8(6) |
y _{ opt } | 2.4(5) | 3.6(4) | 4.6(6) |
Estimated classifier performance
where p _{ T } and p _{ V } are the two prior probabilities (p _{ T }+p _{ V }=1), and η, β, γ are given by Eq. (6). From Eq. (5) it follows that the odds at y=χ are 50:50. By means of Eq. (7), the thresholds χ _{10:90}, χ _{90:10} for the odds 10:90 and 90:10 can be determined. The values of these thresholds in the case of balanced prior probabilities are reported in Table 4.
Bias outliers
In the previous sections, we have addressed the issue of outlier identification by using the triplicate variability. The normality of the “target” and “versus” distributions of the measure of interest y allows for the identification of a second kind of outliers: given a value y, one can promptly evaluate – via Student’s t distribution – the one-tailed probability of obtaining a more extreme value, i.e. a value more displaced by the population mean than the value y. If such p-value is less than a given significance level (for example, 1 %), the value y can be deemed to be a “bias” outlier, i.e. an outlier due to a bias in the triplicate estimates.
Improvement of a classifier’s performance
Looking at Student’s t statistic provides two possible strategies to improve the performance of a classifier. First, one can enhance the difference at the numerator of Student’s t, namely μ _{ T }−μ _{ V }; this solution requires the linear combination of the available class-discriminating measures (in the present case Δ x _{205}) with new, additional measures that also reliably discriminate between the two classes. Such linear combination has to be optimized by means of methods like, for example, principal component analysis or support vector machines. The discussion of these methods goes beyond the goals of the present paper. The second strategy to improve the performance of a classifier consists in reducing the denominator in the expression of Student’s t by linearly combining available measures with new ones. These new measures are not required to be class-discriminating. In the following section the second strategy is analyzed in detail.
Analysis of correlation
Because measure y _{ b } does not discriminate between the two classes, we have μ _{ b,T }≈μ _{ b,V } (the means of the distributions of y _{ b } for the target and the versus class are not significantly different). Consequently, μ _{ T }−μ _{ V }≈μ _{ a,T }−μ _{ a,V }, i.e. nothing significant can be expected with regard to the numerator of Student’s t statistic when μ _{ a } is replaced by μ.
Consequently, the additional measure y _{ b } can be deemed to be a sort of noise-reducer for the class-discriminating measure y _{ a }.
The previous argument still holds if, rather than regarding the whole data set, the correlation appears only on the data subset corresponding to one of the two classes. Therefore, if c is set according to Eq. (9), one of the two standard deviations σ _{ T }, σ _{ V } is reduced whereas the other is enhanced. The net result can be still an increase of the classifier’s performance. The optimal value of c can be assessed by standard analytical and numerical techniques.
Results
A classifier for ADC vs. SQC
Classification | |||||
---|---|---|---|---|---|
Measure | Diagnosis | Target | Versus | ||
ρ>9 | 9>ρ>1 | 9>ρ>1 | ρ>9 | ||
Δ x _{205} | target | 12 | 5 | 1 | 0 |
versus | 0 | 0 | 6 | 9 | |
Δ x _{21} | target | 9 | 9 | 1 | 0 |
versus | 8 | 5 | 2 | 1 | |
−x _{U6} | target | 2 | 14 | 3 | 0 |
versus | 3 | 4 | 8 | 1 | |
y _{ DV } | target | 16 | 1 | 1 | 0 |
versus | 0 | 1 | 1 | 13 | |
y _{ opt } | target | 16 | 1 | 1 | 0 |
versus | 1 | 0 | 0 | 14 |
In the case of the classifier based on Δ x _{205}, by relying on Eqs. (3, 4) a maximum accuracy of 91.4 % ± 3.9 % can be predicted.
Improved classifier for ADC vs. SQC
Pearson correlation coefficient r for the pairs (Δ x _{205}, Δ x _{21}), (x _{205}, x _{U6})
Correlation pair | Set | Size | r | p-value |
---|---|---|---|---|
overall | 33 | -0.02 | 0.92 | |
(x _{205}, x _{U6}) | target | 18 | 0.13 | 0.60 |
versus | 15 | 0.43 | 0.11 | |
overall | 33 | 0.40 | 0.022 | |
(Δ x _{205}, Δ x _{21}) | target | 18 | 0.49 | 0.04 |
versus | 15 | 0.75 | 0.0012 |
Tables 3, 4 and 5 report the statistics of the measure y _{ opt }, and the thresholds and confusion matrix of the classifier relying on this measure, respectively. For the sake of comparison, the same tables also show the data of the classifier relying on the measure y _{ DV } defined in Eq. (2), which was the topic of previous works [13, 17].
In the case of the classifier based on y _{ opt }, by relying on Eqs. (3, 4) a maximum accuracy of 96.1 % ± 2.4 % can be predicted. By comparison, this last parameter is 95.6 % ± 2.6 % in the case of the classifier based on y _{ DV }.
Test of the improved classifier on an independent data set
The miRNA classifier of Eq. (1) provides a different diagnosis than the immunohistochemical analysis only for a single sample. Although this sample does not contain any outlier according to the variability of its triplicates, its value of y _{ opt } appears to be extremely low: according to the statistics of y _{ opt } (see Fig. 3), the probability of getting a more extreme value is p<5·10^{−4}. This hints, rather than to a misclassification of the sample, to a case of “bias” outlier, i.e. to a possibly wrong assessment of the triplicates, as discussed above. For the sake of comparison, for all other 8 samples, as well as for all 33 samples considered in the previous sections, p>0.02
Discussion and conclusion
We introduced a Bayesian classifier that, on the basis of the expression of miR-205, miR-21 and snRNA U6, discriminates samples into two different classes of pulmonary tumors, normally classified by immuno-histochemical approaches: adenocarcinomas and squamous cell carcinomas. The advantage to use miRNAs is due to the ease of their detection and quantification by qRT-PCR, as well as in their extreme specificity. miRNAs are stable molecules well preserved in formalin fixed, paraffin embedded tissues (FFPE) as well as in fresh snap-frozen specimens, unlike larger RNA molecules as messenger RNAs [3].
Our approach is based on a method that employs the quantification of snRNA U6 as a normalizer, miR-21 as a performance enhancer via noise reduction, and miR-205 as a class discriminator. First, we determined that the variance of miRNA quantification triplicates follows a normal chi-square distribution. Thereupon, we designed a procedure to recognize invalid measures (outliers) and remove them from the analysis. The proof that the measures of interest are compatible with normal distributions makes up a crucial step towards the optimization of the Bayesian classifier, the determination of its performance, inclusively of the related uncertainty, and the identification of “bias” outliers. We then proceeded to optimally set our Bayesian classifier and to determine its performance as well as the related uncertainty. Results are displayed in Fig. 2: the classifier based on miR-205 and normalized on snRNA U6 has the best performance.
A main feature of the Bayesian approach described here is the possibility, also in presence of a limited size of the available data sets, of estimating the reliability of a classifier’s performance. This possibility relies on the verification of the normality of the different distributions of interest. Other powerful methods to set up a classifier, such as support vector machines (SVM) and decision trees, though quite versatile to optimize the decisional parameters on a training set, are less suited than a probabilistic approach to provide an immediate quantification of the reliability in the case of application to new sets of data. For example, in the cases discussed above, a SVM approach would likely result in an optimized classifier that also exploits the “apparent” classification capability of Δ x _{21} and even x _{U6} (see, for example, Figs. 1 and 2). However, according to our analysis based on Student’s t, there is no evidence of such capability, so that such a SVM classifier would also possibly have a larger generalization error than a Bayesian classifier of the kind discussed in this paper.
Finally, we provided a method to enhance a classifiers’ performance by exploiting the correlation between the tumor-discriminating miRNA miR-205 and the expression of miR-21, used as a noise reduction factor. The method essentially consists in exploiting the nonzero covariance of two miRNAs, where the first one acts a classifier and the second one is used to abate the variability of the first one. Figure 4 shows the result of an improved classifier, indicating that only 2 samples lay within the uncertainty region, much less than the 12 samples in the case of the non-improved classifier shown in Fig. 2. Results obtained on an independent data set are also satisfactory.
In conclusion, the proposed method introduces a robust tool for determining the cases in which miRNA quantification can be applied in discriminating inter- and intra-tumoral heterogeneity.
Endnote
^{1} Because \({f_{\alpha,\,\nu,\,\infty } = \chi ^{2}_{\alpha,\,\nu }/\nu }\), an alternative outlier definition relying on the F-test would produce the same result as that discussed in this section.
Declarations
Acknowledgements
The authors are grateful to Marco Vardaro for the help in classifying the existing discrimination methods. This project is funded in part by startup funds from the Centre for Integrative Biology (University of Trento) to MAD and by the Outgoing Marie-Curie/Autonomous-Province-of-Trento (PAT) co-fund grant to VDV.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Authors’ Affiliations
References
- Farazi TA, Spitzer JI, Morozov P, Tuschl T. MiRNAs in human cancer. J Pathol. 2011; 223:102–15. doi:10.1002/path.2806.View ArticlePubMedGoogle Scholar
- Del Vescovo V, Grasso M, Barbareschi M, Denti MA. MicroRNAs as lung cancer biomarkers. World J Clin Oncol. 2014; 5:604–20. doi:10.5306/wjco.v5.i4.604.View ArticlePubMedPubMed CentralGoogle Scholar
- Xi Y, Nakajima G, Gavin E, Morris CG, Kudo K, Hayashi K, et al. Systematic analysis of microRNA expression of RNA extracted from fresh frozen and formalin-fixed paraffin-embedded samples. RNA. 2007; 13:1668–1674. doi:10.1261/rna.642907.View ArticlePubMedPubMed CentralGoogle Scholar
- Gilad S, Meiri E, Yogev Y, Benjamin S, Lebanony D, Yerushalmi N, et al. Serum MicroRNAs are promising novel biomarkers. PLoS One. 2008; 3:3148. doi:10.1371/journal.pone.0003148.View ArticleGoogle Scholar
- Grasso M, Piscopo P, Confaloni A, Denti MA. Circulating miRNAs as biomakers for neurodegenerative diseases. Molecules. 2014; 19:6891–910. doi:10.3390/molecules19056891.View ArticlePubMedGoogle Scholar
- Shen J, Stass SA, Jiang F. MicroRNAs as potential biomarkers in human solid tumors. Cancer Lett. 2013; 329:125–36. doi:10.1016/j.canlet.2012.11.001.View ArticlePubMedGoogle Scholar
- Benes V, Castoldi M. Expression profiling of microRNA using real-time quantitative PCR, how to use it and what is available. Methods. 2010; 50:244–9. doi:10.1016/j.ymeth.2010.01.026.View ArticlePubMedGoogle Scholar
- Gallardo E, Navarro A, Vinolas N, Marrades RM, Diaz T, Gel B, et al. miR-34a as a prognostic marker of relapse in surgically resected non-small-cell lung cancer. Carcinogenesis. 2012; 30:1903–1909. doi:10.1093/carcin/bgp219.View ArticleGoogle Scholar
- Landi MT, Zhao Y, Rotunno M, Koshiol J, Liu H, Bergen AW, et al. MicroRNA expression differentiates histology and predicts survival of lung cancer. Clin Cancer Res. 2010; 16:430–41. doi:10.1158/1078-0432.CCR-09-1736.View ArticlePubMedPubMed CentralGoogle Scholar
- Patnaik SK, Kannisto E, Knudsen S, Yendamuri S. Evaluation of microRNA expression profiles that may predict recurrence of localized stage I non-small cell lung cancer after surgical resection. Cancer Res. 2010; 70:36–45. doi:10.1158/0008-5472.CAN-09-3153.View ArticlePubMedGoogle Scholar
- Navarro A, Diaz T, Gallardo E, Vinolas N, Marrades RM, Gel B, et al. Prognostic implications of miR-16 expression levels in resected non-small-cell lung cancer. J Surg Oncol. 2011; 103:411–5. doi:http://dx.doi.org/10.1002/jso.21847.View ArticlePubMedGoogle Scholar
- Gilad S, Lithwick-Yanai G, Barshack I, Benjamin S, Krivitsky I, Edmonston TB, et al. Classification of the four main types of lung cancer using a microRNA-based diagnostic assay. J Mol Diagn. 2012; 14:510–7. doi:10.1016/j.jmoldx.2012.03.004.View ArticlePubMedGoogle Scholar
- Lebanony D, Benjamin H, Gilad S, Ezagouri M, Dov A, Ashkenazi K, et al. Diagnostic assay based on hsa-miR-205 expression distinguishes squamous from nonsquamous non-small-cell lung carcinoma. J Clin Oncol. 2009; 27(12):2030–037. doi:10.1200/JCO.2008.19.4134.View ArticlePubMedGoogle Scholar
- Tan X, Qin W, Zhang L, Hang J, Li B, Zhang C, et al. A 5-microRNA signature for lung squamous cell carcinoma diagnosis and hsa-miR-31 for prognosis. Clin Cancer Res. 2011; 17:6802–811. doi:10.1158/1078-0432.CCR-11-0419.View ArticlePubMedGoogle Scholar
- Lee HW, Lee EH, Ha SY, Lee CH, Chang HK, Chang S, et al. Altered expression of microRNA miR-21, miR-155, and let-7a and their roles in pulmonary neuroendocrine tumors. Pathol Int. 2012; 62:583–91. doi:10.1111/j.1440-1827.2012.02845.x.View ArticlePubMedGoogle Scholar
- Huang W, Hu J, Yang DW, Fan XT, Jin Y, Hou YY, et al. Two microRNA panels to discriminate three subtypes of lung carcinoma in bronchial brushing specimens. Am J Respir Crit Care Med. 2012; 186:1160–1167. doi:10.1164/rccm.201203-0534OC.View ArticlePubMedGoogle Scholar
- Del Vescovo V, Cantaloni C, Cucino A, Girlando S, Silvestri M, Bragantini E, et al. miR-205 expression levels in nonsmall cell lung cancer do not always distinguish adenocarcinomas from squamous cell carcinomas. Am J Surg Pathol. 2011; 35(2):268–75. doi:10.1097/PAS.0b013e3182068171.View ArticlePubMedGoogle Scholar
- Peltier HJ, Latham GJ. Normalization of microRNA expression levels in quantitative RT-PCR assays: identification of suitable reference RNA targets in normal and cancerous human solid tissues. RNA. 2008; 14:844–52. doi:10.1261/rna.939908.View ArticlePubMedPubMed CentralGoogle Scholar
- Vilardi A, Tabarelli D, Ricci L. Tailoring a psychophysical discrimination experiment upon assessment of the psychometric function: Predictions and results. AIP Adv. 2015; 5:027121. doi:10.1063/1.4908271.View ArticleGoogle Scholar
- Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinforma. 2011; 12:77. doi:10.1186/1471-2105-12-77.View ArticleGoogle Scholar