Exposure time independent summary statistics for assessment of drug dependent cell line growth inhibition
- Steffen Falgreen^{1}Email author,
- Maria Bach Laursen^{1},
- Julie Støve Bødker^{1},
- Malene Krag Kjeldsen^{1},
- Alexander Schmitz^{1},
- Mette Nyegaard^{2},
- Hans Erik Johnsen^{1},
- Karen Dybkær^{1} and
- Martin Bøgsted^{1, 3}
DOI: 10.1186/1471-2105-15-168
© Falgreen et al.; licensee BioMed Central Ltd. 2014
Received: 15 May 2013
Accepted: 27 May 2014
Published: 5 June 2014
Abstract
Background
In vitro generated dose-response curves of human cancer cell lines are widely used to develop new therapeutics. The curves are summarised by simplified statistics that ignore the conventionally used dose-response curves’ dependency on drug exposure time and growth kinetics. This may lead to suboptimal exploitation of data and biased conclusions on the potential of the drug in question. Therefore we set out to improve the dose-response assessments by eliminating the impact of time dependency.
Results
First, a mathematical model for drug induced cell growth inhibition was formulated and used to derive novel dose-response curves and improved summary statistics that are independent of time under the proposed model. Next, a statistical analysis workflow for estimating the improved statistics was suggested consisting of 1) nonlinear regression models for estimation of cell counts and doubling times, 2) isotonic regression for modelling the suggested dose-response curves, and 3) resampling based method for assessing variation of the novel summary statistics. We document that conventionally used summary statistics for dose-response experiments depend on time so that fast growing cell lines compared to slowly growing ones are considered overly sensitive. The adequacy of the mathematical model is tested for doxorubicin and found to fit real data to an acceptable degree. Dose-response data from the NCI60 drug screen were used to illustrate the time dependency and demonstrate an adjustment correcting for it. The applicability of the workflow was illustrated by simulation and application on a doxorubicin growth inhibition screen. The simulations show that under the proposed mathematical model the suggested statistical workflow results in unbiased estimates of the time independent summary statistics. Variance estimates of the novel summary statistics are used to conclude that the doxorubicin screen covers a significant diverse range of responses ensuring it is useful for biological interpretations.
Conclusion
Time independent summary statistics may aid the understanding of drugs’ action mechanism on tumour cells and potentially renew previous drug sensitivity evaluation studies.
Keywords
Dose response experiments NCI60 Doxorubicin Mathematical modelling Differential equation modelling Nonlinear regression Isotonic regression Bootstrap Parametric bootstrapBackground
An essential part of discovery and development of anticancer drugs is to assess the induced growth inhibition in a biologically broad range of tumour derived cell lines by dose-response experiments [1, 2]. The three large cell line screens NCI60 [3, 4], JFCR39 [5, 6], and CMT1000 [2, 7] are among the most well-known high throughput cell line drug screens.
The approach used in CMT1000 and several other studies [8–10] is currently the standard approach for conducting dose-response experiments. The experiments are performed by challenging exponentially growing cell lines with a serial dilution of drug concentrations and estimating growth inhibition by relative cell counts between the treated and untreated cell line. Then, a summary statistic of drug efficiency ${\mathit{\text{GI}}}_{50}^{R}$ (50% growth inhibition) is obtained by estimating the concentration at which the relative cell count is 50% after a fixed period of time. Hence, neither drug exposure time nor varying cell line growth rates are considered.
Panel C illustrates dose-response curves calculated at three time points: 25, 49, and 73 hours, for the two cell line models. Because of the fast growth rate of cell line model 2, the summary statistic ${\mathit{\text{GI}}}_{50}^{R}$ is obtained at a lower concentration for this cell line model than for cell line model 1 for each of the three time points. This indicates that cell line model 2 is evaluated as the more sensitive of the two. Hence, this assessment of growth inhibition generates summary statistics that are incomparable between cell lines with different growth rates.
The dose-response experiments performed for the NCI60 and JFCR39 screens are summarised by comparing net differences between cell counts at observation time and the initial cell counts for the treated and untreated cell lines. As we illustrate later this method only partially solves the problem of growth rate dependency.
The concept behind the present work is that modelling the growth of a cell line exposed to a drug by a simplified differential equation will allow us to derive dose-response curves and summary statistics that are independent of time under the proposed model. For estimation of the improved summary statistics a statistical workflow is suggested consisting of 1) pre-processing of absorbance measurements to account for multiplicative errors originating from e.g. cell line seeding [11] and correcting for background absorbance caused by the drug [12], 2) isotonic regression for modelling the dose-response curve which is robust against outliers and model misspecifications [13, 14], and 3) a bootstrap method for estimation of confidence intervals for summary statistics [9]. We also aim to illustrate a transformation of the model used in the cell line screen NCI60, which accounts for each cell line’s doubling time and enables a reanalysis of existing dose-response data.
Finally, the adequacy of the differential equation for modelling real data is tested using a doxorubicin screen. The screen is also used to investigate the applicability of the proposed statistical analysis workflow by providing variance estimates for obtained exposure time independent summary statistics.
Methods
The mathematical model
and T_{ c }corresponds to the net observed doubling or halving time at concentration c.
where GI_{50} (50% growth inhibition) denotes the concentration at which the cell line grows with a doubling time twice as long as the same cell line untreated, TGI (total growth inhibition) denotes the concentration at which the cell line has no net growth, and L C_{ t }(lethal concentration t) denotes the concentration at which the cell count decays with a halving time of t hours. For example LC_{48} is the concentration at which N (48,c) = N_{0}/2.
The growth inhibition induced by these drug concentrations is illustrated in Figure 2C for a cell line model with doubling time T_{0} = 60 hours and N_{0} = 30,000. At the concentration corresponding to GI_{50} the doubling time for the cell line is doubled to 120, TGI the halving time ${T}_{c}^{\u2020}={T}_{0}=60$ such that the growth of the cell line is completely halted, and LC_{48} the halving time for the cell line is 48 hours.
It is noteworthy that the G-model is independent of the duration of the dose-response experiment. The model is summarised by the statistics GI_{50}, TGI, and LC_{48} at which the G (t,c) equals 0.5, 0, and -1/48, respectively. In general we define GI_{ x }and LC_{ t }to be the concentrations where G (t,c) = (100-x)/100 and G (t,c) = -1/t.
The cell counts which the D-model is based upon are illustrated by the triangle and circles in Figure 2C for t = 48 hours. For this model x % growth inhibition $G{I}_{x}^{D}$ and y % lethal concentration $L{C}_{y}^{D}$ are attained at concentrations c_{1} and c_{2} where D(t,c_{1}) = (100-x)/100 and D (t,c_{2}) = -(100-y)/100. The dose-response model is usually summarised for a fixed t by ${\mathit{\text{GI}}}_{50}^{D}$, ${\mathit{\text{LC}}}_{50}^{D}$, and TGI^{ D }the latter of which is attained at the concentration c where D (t,c) = 0.
The cell counts which the R-model is based upon are illustrated by circles in Figure 2C for t = 48 hours. For this model x % growth inhibition $G{I}_{x}^{R}$ is attained at concentration c where R (t,c) = (100-x)/100. The R-model is usually summarised by ${\mathit{\text{GI}}}_{25}^{R}$, ${\mathit{\text{GI}}}_{50}^{R}$, and ${\mathit{\text{GI}}}_{75}^{R}$.
For a fixed t, the graph of a dose-response model, say G, {(c,G (t,c)):c > 0} is denoted the dose-response curve of G. As the D- and R-models suggest, the corresponding dose-response curves are dependent on the time t, whereas the dose-response curve of G is not.
Notice it is possible to define a fourth summary statistic AUC_{ q }(area under curve) which is the area above a specified value q and below the dose-response curve [16]. Thus, for the dose-response models G and D, AUC_{0} is the area under the dose-response curve for which the cell count is still increasing with time.
Note, however, that both transformations require access to the cell line specific doubling time T_{0}.
Estimation of cell count
Absorbance measurements can be utilised as surrogates for the cell count N(t,c) and thereby used to estimate the three dose-response models. This is generally done using an MTS assay (3-(4,5-dimethylthiazol-2-yl)-5-(3-carboxymethoxyphenyl)-2-(4-sulfophenyl)-2H-tetrazolium) that exploits the mitochondrial reduction of tetrazolium to an aqueous soluble formazan product by the dehydrogenase enzyme in viable cells at 37°C. The amount of produced formazan is directly proportional to the cell count N (t,c) and can be quantified colourimetrically by measuring absorbance at 492 nm [12].
where γ is a proportionality factor and α_{ ti }is the absorbance at time t, for a cell line exposed to drug concentration c_{ i }, i = 1,…,I where 0 = c_{0} < c_{1} < ⋯ < c_{ I }. The proportionality factor is cell line specific due to individual capabilities of reducing tetrazolium into the coloured formazan product.
where δ_{ kt }is the inter-plate variation assumed to be multiplicative, β_{ kt }is the plate specific background absorbance which is assumed to be additive, and, finally, the technical variation ε_{ ktil }is assumed to be additive and normally distributed with mean zero and following a heteroscedastic variance model $|{\delta}_{\mathit{\text{kt}}}{\alpha}_{\mathit{\text{ti}}}+{\beta}_{\mathit{\text{kt}}}{|}^{2\xi}{\sigma}_{\beta}^{2}$.
Statistical analysis workflow
The proposed statistical workflow has been implemented in the statistical software R version 3.0.1. The parameter estimation is performed using the function gnls from the package nlme[17]. Isotonic regression is implemented by the function isoreg from the library stats, and the area under the curve is calculated by the function trapz from the library pracma.
Model-based pre-processing
The nonlinear regression model corresponds to the formalism in Chapter 7 of [17] and can be estimated by the methods herein. The components of the heteroscedastic variance is estimated by an iteratively reweighted scheme, see page 207 of [17]. The I + 1 coefficients ${\hat{\alpha}}_{\mathit{\text{ti}}}$ are the summarised absorbance measures for each concentration c_{ i }. Since negative absorbance measures are meaningless, all absorbance estimates below a pre-specified value are replaced by this value. We use the value 0.025 as the cut point in the current study.
One of the favourable features of the model-based approach to pre-processing dose-response experiments is outlier detection based on residuals. The residuals are the difference between the observed values and the values estimated by the regression model. First, the regression model is fitted to all data. Absorbance measures with residuals greater than a pre-specified number of standard deviations are regarded as outliers and removed. Based on the remaining absorbance measures the model is fitted again and outliers are detected and removed. This process is iterated until no outliers are detected or until a pre-specified maximum number of iterations is reached. In the current study we use 3 standard deviations and iterate the process twice.
Estimation of cell line doubling times
for i = 1,…,I.
Estimation of the dose-response curve
The dose-response model R is estimated pointwise by (7) with $\hat{N}(t,c)={\hat{\alpha}}_{\mathit{\text{tc}}}$. The dose-response curve for the R-model $\hat{R}(t,c)$ is obtained using isotonic regression and linear interpolation between the pointwise estimates.
Estimation of summary statistics
- 1.
For j in 1:J
- 1)
Generate K plate sets on basis of the pre-processing model fitted to each cell line.
- 2)
Fit the pre-processing model without outlier detection.
- 3)
Fit the growth model to the pre-processed absorbance measurements.
- 4)
Calculate the growth inhibition on basis of the G-model.
- 5)
Estimate the summary statistics GI _{50}, TGI, LC _{48}, and AUC _{0}.
- 2.
Estimate a confidence interval for each summary statistic by use of the 2.5% and 97.5% percentiles of the J estimates obtained in step 5).
A similar approach can be used to estimate summary statistics for the dose-response models D and R with summary statistics obtained by the concentrations where $\hat{D}(t,c)$ and $\hat{R}(t,c)$ equal e.g. 0.5, 0, and -0.5 and 0.75, 0.5, and 0.25, respectively.
Correction of background absorbance
When the drug under investigation is coloured like e.g. doxorubicin or interacts with the MTS assay, the background absorbance measures are elevated for increasing drug concentrations. This elevation necessitates correction when estimating the cell count [12]. One method is to include a background control for each concentration of the drug. Such an approach, however, requires a large number of wells. Alternative one may create a number of background plates with a setup similar to the one used for evaluating the cell count but without seeding cells into them. Next, these plates are pre-processed as described in Model-based pre-processing, which results in measurements of the absorbance caused by the drug. Finally, the excessive absorbance caused by each concentration of the drug is subtracted from the raw absorbance. This is done as an initial step before the Statistical analysis workflow.
The simulation study
Characteristics of the used cell line models
Cell line | T _{0} | GI _{50} | TGI | LC _{48} |
---|---|---|---|---|
Cell 1 | 60 | -8.40 | -8.03 | -7.57 |
Cell 2 | 30 | -8.18 | -7.78 | -7.47 |
Cell 3 | 15 | -7.95 | -7.49 | -7.27 |
Cell 4 | 15 | -7.72 | -7.27 | -7.05 |
Cell 5 | 30 | -7.50 | -7.11 | -6.80 |
Cell 6 | 60 | -7.28 | -6.91 | -6.44 |
Cell 7 | 15 | -7.05 | -6.59 | -6.37 |
Cell 8 | 60 | -6.82 | -6.46 | -5.99 |
Cell 9 | 30 | -6.60 | -6.21 | -5.90 |
where a = 2, b = 1/5, d = 0, f = 1. The parameter e is specified such that the summary statistics shown in Table 1 are obtained.
Each cell line model is simulated with N_{0} = 10,000 cells seeded into each well for all four plates at time t = 0 together with the drug. In order to imitate laboratory conditions the MTS assay is assumed added at time points ${t}_{1}^{\prime}=0,\phantom{\rule{1em}{0ex}}{t}_{2}^{\prime}=24,\phantom{\rule{1em}{0ex}}{t}_{3}^{\prime}=48,\text{and}\phantom{\rule{0.3em}{0ex}}{t}_{4}^{\prime}=72$ hours and the absorbance is measured after t_{ inc }= 2 hours. Consequently, the cell counts are generated at ${t}_{1}={t}_{1}^{\prime}+{t}_{\mathit{\text{inc}}}/2=1,\phantom{\rule{1em}{0ex}}{t}_{2}=25,\phantom{\rule{1em}{0ex}}{t}_{3}=49,\text{and}\phantom{\rule{0.3em}{0ex}}{t}_{4}=73$. In order to attain absorbance measurements the proportionality factor γ in (10) is set equal to 0.4/10,000 for all nine cell line models.
In order to investigate the dose-response model G’s capability of estimating the summary statistics GI_{50}, TGI, and L C_{48}, noise is added to the system. To mimic real data this is done according to absorbance model (11) with parameters chosen in concordance with the estimates obtained for the B-cell cancer cell line panel introduced later.
For each cell line model the plate specific background absorbance β_{ kt }is drawn from a lognormal distribution with mean μ_{ β }= -0.8 and standard deviation σ_{ β }= 0.13; the plate specific multiplicative error δ_{ kt }is likewise drawn from a lognormal distribution with mean μ_{ δ }= 0 and standard deviation σ_{ δ }= 0.38. The parameters μ_{ β }, μ_{ δ }, σ_{ β }, and σ_{ δ }are respectively chosen as the mean and standard deviation of the log transformed estimates for β and δ obtained for the B-cell cancer cell line panel. Finally, the technical variation ε_{ ktcl }is drawn from a mean zero normal distribution with heteroscedastic variance $|{\delta}_{\mathit{\text{kt}}}{\alpha}_{\mathit{\text{tc}}}+{\beta}_{\mathit{\text{kt}}}{|}^{2\xi}{\sigma}_{\u03f5}^{2}$ where ξ = 1.42 and σ_{ ε }= 0.074 are chosen as the medians of the estimates for ξ and σ_{ ε }obtained for the B-cell cancer cell line panel.
The Statistical analysis workflow is used to obtain estimates of the summary statistics GI_{50}, TGI, and LC_{48} associated with each cell line for 1,000 simulated datasets. Finally, the mean bias, standard deviation, and mean square error (MSE) are calculated for each cell line model and time point.
The NCI60 cancer cell line panel
The cell line screen NCI60 is utilised to quantify the effect of a cell line’s doubling time on the GI_{50}-value obtained by the dose-response model D in real data. Pharmacological data generated in the screen and modelled by the D-model is available online for 49,450 different compounds: http://dtp.nci.nih.gov. In this study we apply all compounds available in the September 2012 release that are tested at least three times on more than half the cell lines and for which half of the tested cell lines are affected by the drug. These criteria are satisfied for 1,699 different compounds.
The growth inhibition data is averaged for all experiments that are not already summarised by the mean. Next, the G-model is calculated by use of the transformation (8). For the dose-response models G and D the summary statistics GI_{50} and ${\mathit{\text{GI}}}_{50}^{D}$ are estimated using isotonic regression.
The association between the cell lines’ doubling time T_{0} and the summary statistics GI_{50} and ${\mathit{\text{GI}}}_{50}^{D}$ is determined using Pearson’s correlation coefficient for all 1,699 compounds. In order to determine whether or not the transformation causes a significant reduction in the correlation a paired t-test is used.
The reduction in the aforementioned correlation is further illustrated for doxorubicin and the drug with the greatest change. For these compounds the summary statistics GI_{50} and ${\mathit{\text{GI}}}_{50}^{D}$ are plotted against the doubling time T_{0}, and the Pearson’s correlation coefficient is calculated.
The B-cell cancer cell line panel
A doxorubicin dose-response screen of 14 Diffuse Large B-cell Lymphoma and 12 Multiple Myeloma cell lines is used to illustrate the proposed Statistical analysis workflow. The origin of the cell lines is as listed: KMM-1 and KMS-11 were obtained from JCRB (Japanese Collection of Research Bioresources). AMO-1, DB, KMS-12-PE, KMS-12-BM, LP-1, MC-116, MOLP-8, NCI-H929, NU-DHL-1, NU-DUL-1, OPM-2, RPMI-8226, SU-DHL-4, SU-DHL-5, and U-266 were purchased from DSMZ (German Collection of Microorganisms and Cell Cultures). FARAGE, HBL-1, OCI-Ly3, OCI-Ly7, OCI-Ly19, RIVA, SU-DHL-8, and U2932 were kindly provided by Dr. Jose A. Martinez-Climent (Molecular Oncology Laboratory University of Navarra, Pamplona, Spain). Finally, Dr. Steven T. Rosen generously provided MM1S.
The identity of the cell lines was verified by DNA barcoding performed every time a cell line was thawed and brought into culture. In brief, PCR analysis of left over traces of genomic DNA in 0.2 ng/μl extracted RNA from cell lines was used as template in a multiplex PCR amplifying specific repeated DNA regions using the AmpFISTR Identifiler PCR amplification kit (Applied Biosystems, CA, USA). A fragment analysis of the amplified PCR products was performed by capillary electrophoresis (Eurofins Medigenomix GmbH, Applied Genetics, Germany). The resulting FSA file was analysed using the Osiris software (http://ncbi.nlm.nih.gov/projects/SNP/osiris) confirming the identity of the cell lines.
B-cell cancer cell lines and culture conditions
The cell lines were cultured under standard conditions at 37° C; in a humidified atmosphere of 95% air and 5% CO _{2} with the appropriate medium (RPMI1640 or IMDM), fetal bovine serum (FBS), and 1% penicillin/streptomycin. The cell lines were maintained for a maximum of 20 passages to minimize any long-term culturing effects. Penicillin/streptomycin 1%, RPMI1640, IMDM and FBS were purchased from Invitrogen.
Dose-response experiments
Doxorubicin is a coloured agent which was accounted for according to Correction of background absorbance. Using the corrected absorbance measurements, the dose-response model G and time independent summary statistics GI_{50}, TGI, LC_{48}, and AUC_{0} were estimated according to the established Statistical analysis workflow. The outlined bootstrap algorithm was applied to estimate 95% confidence intervals for the summary statistics with the number of iterations J = 200.
Model check
Since different drugs have different action mechanisms one should investigate whether or not the proposed differential equation models data adequately well. This is, however, not possible if the dose response experiment has not been conducted for more than two time points. Here the experiment was expanded to include five time points ${t}_{1}^{\prime}=0$, ${t}_{2}^{\prime}=12$, ${t}_{3}^{\prime}=24$, ${t}_{4}^{\prime}=36$, ${t}_{5}^{\prime}=48$. Each of the five plates are configured with the same setup as that described in section Dose-response experiments with t_{ inc }= 2 hours. This approach gives estimates of the cell counts at approximately t_{1} = 1, t_{2} = 13, t_{3} = 25, t_{4} = 37, t_{5} = 49 hours. The experiment was repeated thrice. The differential equation model is fitted to the data using only t_{1} and t_{5}. By plotting the data for all time points together with the fitted model it is possible to observe whether or not the model fits adequately well or whether a more advanced model is necessary.
Results
The simulation study
The effect of the drug was modelled to be constant through time. However, according to both the R- and D-models the cell lines’ sensitivity toward the drug increased with time.
More specifically, for the R-model this is illustrated in Panels A, D, and G of Figure 5 which depict the obtained dose-response curves for the three time points. For each cell line model the dose-response curves had the same sigmoidal shape for all time points. As shown in (7) the R-model is indifferent towards the cell line doubling time which entailed that the drug sensitivity increased equivalently for each cell line model as the drug exposure time was prolonged. Furthermore, cell line model 1 was simulated as the most sensitive with a GI_{50}-value of -8.4 log10(mmol/ml). However, for all time points, the R-model evaluated it as the fourth most sensitive, surpassed by cell line models 2, 3, and 4 which were simulated with GI_{50}-values of -8.18, -7.95, and, -7.72 log10(mmol/ml), respectively.
For the D-model the increase in sensitivity was related to the growth rate of the cell line such that the order of the cell lines’ sensitivity interchanged through time. This is illustrated in Panels B, E, and H where the dose-response curves obtained by the D-model are shown for the three time points. The cell line models 3, 4, and 7 and 1, 6, and 8 were respectively fast and slowly growing. Accordingly, the increase in sensitivity through time was much more pronounced for cell line models 3, 4, and 7 than for 1, 6, and 8. In particular the fast growth rate of cell line model 7 and slow growth rate of cell line model 6 caused the obtained ${\mathit{\text{GI}}}_{50}^{D}$-values to interchange throughout the three time points.
The cell lines 1, 2, and 3 were simulated with GI_{50}-values of -8.40, -8.18, and, -7.95 log10(mmol/ml), respectively; however, the ${\mathit{\text{GI}}}_{50}^{D}$-values obtained by the D-model were indistinguishable in Panel H. Additionally, the sensitivity level for cell line models 6 and 7 were reversed such that cell line 7 was evaluated to be more sensitive to the drug than cell line 6.
This implied that the summary statistics obtained by the R- and D-models were biased and relative to the cell lines’ sensitivity they were ordered incorrectly. Time independent summary statistics obtained by the G-model equalled those shown in Table 1.
The dose-response models G and D are continuous everywhere and differentiable everywhere except at the TGI-value. The latter results in the singularity occurring at that value for both functions. Since the R-model is continuous and differentiable everywhere such singularities do not occur for this model.
Summary of the simulation study
Mean bias | Standard deviation | MSE | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
GI _{50} | TGI | LC _{48} | GI _{50} | TGI | LC _{48} | GI _{50} | TGI | LC _{48} | |||
Cell line 1, T _{ 0 } = 60 | |||||||||||
Time 25 | 0.01 | 0.04 | 0.07 | 0.49 | 0.49 | 0.22 | 0.24 | 0.24 | 0.05 | ||
Time 49 | -0.07 | -0.06 | 0.00 | 0.36 | 0.36 | 0.14 | 0.13 | 0.14 | 0.02 | ||
Time 73 | -0.05 | -0.05 | -0.01 | 0.25 | 0.25 | 0.11 | 0.06 | 0.07 | 0.01 | ||
Cell line 2, T _{ 0 } = 30 | |||||||||||
Time 25 | -0.10 | -0.08 | -0.01 | 0.41 | 0.41 | 0.24 | 0.17 | 0.17 | 0.06 | ||
Time 49 | -0.04 | -0.04 | -0.02 | 0.18 | 0.19 | 0.13 | 0.03 | 0.04 | 0.02 | ||
Time 73 | -0.02 | -0.02 | -0.01 | 0.11 | 0.12 | 0.09 | 0.01 | 0.01 | 0.01 | ||
Cell line 3, T _{ 0 } = 15 | |||||||||||
Time 25 | -0.05 | -0.04 | -0.02 | 0.21 | 0.25 | 0.21 | 0.05 | 0.06 | 0.05 | ||
Time 49 | -0.01 | -0.01 | 0.00 | 0.08 | 0.11 | 0.10 | 0.01 | 0.01 | 0.01 | ||
Time 73 | -0.02 | -0.01 | 0.00 | 0.07 | 0.09 | 0.08 | 0.01 | 0.01 | 0.01 | ||
Cell line 4, T _{ 0 } = 15 | |||||||||||
Time 25 | -0.02 | 0.00 | 0.01 | 0.20 | 0.22 | 0.20 | 0.04 | 0.05 | 0.04 | ||
Time 49 | -0.01 | 0.00 | 0.00 | 0.09 | 0.11 | 0.10 | 0.01 | 0.01 | 0.01 | ||
Time 73 | -0.04 | -0.04 | -0.03 | 0.10 | 0.11 | 0.11 | 0.01 | 0.01 | 0.01 | ||
Cell line 5, T _{ 0 } = 30 | |||||||||||
Time 25 | -0.08 | -0.06 | 0.00 | 0.43 | 0.43 | 0.23 | 0.19 | 0.19 | 0.05 | ||
Time 49 | -0.03 | -0.03 | -0.01 | 0.17 | 0.18 | 0.12 | 0.03 | 0.03 | 0.01 | ||
Time 73 | -0.03 | -0.03 | -0.01 | 0.12 | 0.13 | 0.09 | 0.02 | 0.02 | 0.01 | ||
Cell line 6, T _{ 0 } = 60 | |||||||||||
Time 25 | -0.07 | 0.00 | 0.07 | 0.69 | 0.63 | 0.23 | 0.47 | 0.40 | 0.06 | ||
Time 49 | -0.08 | -0.06 | 0.00 | 0.41 | 0.38 | 0.14 | 0.18 | 0.15 | 0.02 | ||
Time 73 | -0.04 | -0.03 | 0.00 | 0.25 | 0.23 | 0.10 | 0.06 | 0.06 | 0.01 | ||
Cell line 7, T _{ 0 } = 15 | |||||||||||
Time 25 | -0.05 | -0.03 | -0.02 | 0.22 | 0.24 | 0.20 | 0.05 | 0.06 | 0.04 | ||
Time 49 | -0.02 | -0.01 | 0.00 | 0.09 | 0.11 | 0.10 | 0.01 | 0.01 | 0.01 | ||
Time 73 | -0.10 | -0.11 | -0.04 | 0.22 | 0.23 | 0.10 | 0.06 | 0.06 | 0.01 | ||
Cell line 8, T _{ 0 } = 60 | |||||||||||
Time 25 | -0.10 | -0.03 | 0.06 | 0.73 | 0.65 | 0.22 | 0.54 | 0.42 | 0.05 | ||
Time 49 | -0.14 | -0.09 | -0.02 | 0.49 | 0.41 | 0.15 | 0.26 | 0.18 | 0.02 | ||
Time 73 | -0.06 | -0.05 | -0.01 | 0.29 | 0.27 | 0.11 | 0.09 | 0.08 | 0.01 | ||
Cell line 9, T _{ 0 } = 30 | |||||||||||
Time 25 | -0.09 | -0.06 | 0.00 | 0.46 | 0.44 | 0.23 | 0.22 | 0.20 | 0.05 | ||
Time 49 | -0.04 | -0.03 | -0.01 | 0.17 | 0.19 | 0.12 | 0.03 | 0.04 | 0.02 | ||
Time 73 | -0.02 | -0.02 | -0.01 | 0.11 | 0.12 | 0.08 | 0.01 | 0.01 | 0.01 |
Model check
The NCI60 cancer cell line panel
In Figure 7B to 7E the considered correlation is illustrated for the single agent doxorubicin and the drug giving rise to the greatest change in correlation by plotting the summary statistics ${\mathit{\text{GI}}}_{50}^{D}$ and GI_{50} against T_{0}. For doxorubicin the correlation was 0.16, (95% CI: (-0.1,0.4)) for the uncorrected D-model and -0.03, (95% CI: (-0.28,0.22)) for the G-model. The drug with NSC number 624806 gave rise to the greatest change in correlation, specifically, from 0.41, (95% CI: (0.15,0.62)) for the uncorrected D-model to -0.26, (95% CI: (-0.5,0.02)) for the G-model.
The ten drugs with the greatest negative correlation between T_{0} and GI_{50} for model G have NSC numbers: 38721, 343513, 338720, 638850, 637404, 624807, 59729, 630684, 698673, and 353882. For model D the drugs were: 637404, 638850, 19994, 698673, 627666, 637603, 626734, 630684, 690134, and 37364. Out of these drugs four were found through both model G and D.
The B-cell cancer cell line panel
In Figure 9 the result of the pre-processing procedure is illustrated for the cell line SU-DHL-4. Panels A and B show the raw absorbance measures for the four replicated experiments whereas the effect of the colour correction is shown in Panels C and D. In Panels E and F the results of the conventionally applied background correction are depicted. Finally, the result of the Model-based pre-processing is illustrated in Panels G and H. When comparing panels E and F to G and H we noticed that the mean absorbance was estimated with a considerable lower variance when the model-based pre-processing was used. A cross marks the measurements that are found to be outliers and for example two of the un-treated control measurements for plate 2 were deemed to be outliers as illustrated in Panel H. In panel H these measurements were clearly extreme values, however, prior to the model-based pre-processing this was not the case.
Summary statistics for the B-cell cancer cell line panel
Cell Line | T _{0} | G-Model | ||||
---|---|---|---|---|---|---|
Hours | GI _{50} | TGI | LC _{48} | AUC _{0} | ||
DLBCL | ||||||
OCI-Ly7 | 24 (21;26) | -6.23 (-6.31;-6.18) | -5.96 (-6.05;-5.84) | -5.57 (-5.74;-5.31) | 325 (310;332) | |
RIVA | 68 (55;91) | -6.35 (-6.39;-6.27) | -5.87 (-6.18;-5.79) | -4.97 (-5.36;-4.82) | 327 (316;337) | |
U2932 | 34 (33;35) | -6.53 (-6.56;-6.51) | -6.28 (-6.29;-6.25) | -5.60 (-5.68;-5.48) | 295 (285;299) | |
DB | 37 (35;39) | -6.58 (-6.65;-6.52) | -6.19 (-6.25;-6.15) | -5.71 (-5.76;-5.65) | 276 (260;285) | |
SU-DHL-4 | 36 (34;39) | -6.59 (-6.64;-6.52) | -6.33 (-6.39;-6.24) | -5.92 (-6.05;-5.73) | 289 (277;294) | |
MC-116 | 50 (41;70) | -6.64 (-6.87;-6.48) | -6.29 (-6.38;-6.20) | -5.57 (-5.62;-5.53) | 272 (233;300) | |
HBL-1 | 46 (42;51) | -6.83 (-6.89;-6.78) | -6.46 (-6.55;-6.37) | -5.76 (-5.83;-5.68) | 272 (266;277) | |
OCI-Ly3 | 91 (69;119) | -7.02 (-7.06;-6.94) | -6.89 (-6.96;-6.81) | -6.67 (-6.73;-6.59) | 253 (244;259) | |
NU-DUL-1 | 48 (40;63) | -7.27 (-7.33;-7.21) | -7.13 (-7.14;-7.11) | -7.02 (-7.04;-7.00) | 224 (191;231) | |
SU-DHL-8 | 85 (53;175) | -7.32 (-7.39;-7.21) | -7.19 (-7.32;-7.10) | -7.01 (-7.07;-6.93) | 222 (207;231) | |
NU-DHL-1 | 29 (27;32) | -7.39 (-7.49;-7.29) | -6.94 (-6.98;-6.89) | -6.66 (-6.70;-6.63) | 201 (184;213) | |
SU-DHL-5 | 32 (30;35) | -7.42 (-7.44;-7.40) | -7.29 (-7.32;-7.26) | -7.12 (-7.16;-7.08) | 202 (189;209) | |
FARAGE | 58 (49;72) | -7.79 (-7.93;-7.49) | -6.93 (-7.23;-6.80) | -6.44 (-6.52;-6.37) | 180 (158;198) | |
OCI-Ly19 | 39 (36;45) | -7.87 (-7.95;-7.79) | -7.13 (-7.25;-6.83) | -6.39 (-6.46;-6.31) | 167 (157;178) | |
MM | ||||||
KMS-11 | 48 (45;51) | -5.91 (-5.94;-5.88) | -5.65 (-5.68;-5.63) | -5.46 (-5.48;-5.44) | 356 (340;361) | |
KMM-1 | 44 (40;52) | -6.03 (-6.13;-5.95) | -5.87 (-5.91;-5.85) | -5.70 (-5.74;-5.66) | 346 (316;350) | |
KMS-12-PE | 24 (20;29) | -6.04 (-6.24;-5.91) | -5.73 (-5.82;-5.61) | -5.50 (-5.61;-5.32) | 340 (310;353) | |
LP-1 | 33 (30;36) | -6.07 (-6.12;-6.02) | -5.82 (-5.85;-5.80) | -5.60 (-5.62;-5.59) | 316 (294;336) | |
U-266 | 48 (43;54) | -6.13 (-6.20;-6.06) | -5.86 (-5.91;-5.80) | -5.42 (-5.47;-5.37) | 325 (299;339) | |
OPM-2 | 57 (47;71) | -6.24 (-6.33;-6.13) | -5.98 (-6.07;-5.91) | -5.61 (-5.64;-5.57) | 329 (313;339) | |
KMS-12-BM | 47 (42;57) | -6.25 (-6.33;-6.16) | -5.92 (-6.02;-5.85) | -5.59 (-5.61;-5.56) | 331 (307;337) | |
RPMI-8226 | 30 (29;32) | -6.42 (-6.45;-6.40) | -6.11 (-6.14;-6.09) | -5.87 (-5.89;-5.85) | 297 (292;306) | |
NCI-H929 | 28 (25;31) | -6.50 (-6.54;-6.48) | -6.39 (-6.42;-6.34) | -6.25 (-6.31;-6.17) | 291 (270;300) | |
AMO-1 | 32 (30;34) | -6.74 (-6.76;-6.72) | -6.61 (-6.64;-6.58) | -6.30 (-6.40;-6.17) | 269 (265;283) | |
MOLP-8 | 34 (21;48) | -6.78 (-6.93;-6.62) | -6.52 (-6.67;-6.27) | -6.23 (-6.37;-6.10) | 253 (229;290) | |
MM1S | 37 (26;51) | -7.20 (-7.29;-7.10) | -7.05 (-7.12;-6.95) | -6.87 (-6.94;-6.60) | 227 (208;239) |
Discussion
In the present study a differential equation that models drug induced growth inhibition of human tumour cell lines was established. Based on this equation a novel model for summarising dose-response experiments was produced, that in combination with a statistical workflow, is capable of generating unbiased time independent summary statistics.
To determine if the differential equation is adequate for modelling real data a time experiment based on doxorubicin was conducted. The experiment included five time points of which only the first and last were used to fit the differential equation. The differential equation was found to model the data adequately, albeit the use of only two time points may lead to an underestimated drug efficiency for large doses. Since the differential equation was found adequate, a simulation study was performed to document the potential bias when using existing methods, and the robustness of the workflow. We deduced that under the proposed differential equation these summary statistics are biased estimators of growth inhibition so that the drug effect is amplified concurrently with increasing growth rate of the cell lines.
In Kondoh et al. [18] 40 representative anticancer drugs from the NCI60 screen were used to illustrate the association between cell line growth rate and drug sensitivity assessed by the D-model. They found the growth rate was positively correlated with drug sensitivity. We propose that this finding is partly caused by systematic bias induced by the experimental setup of the cell line screens. Since the difference between the treated and un-treated cell line will increase with time, the effect of the drug will seem greater for fast growing cell lines. We showed that by transforming data obtained by the D-model into the G-model the correlation between doubling time and drug resistance decreased significantly. We do not argue against drug resistance being associated with growth rate as the authors successfully discover and validate a potential new anticancer drug, we merely suggest that removing the design-based bias may lead to a range of new potential drugs to be investigated.
In order to illustrate the suggested workflow for dose-response experiments, a study of 26 cell lines tested for drug resistance at 18 different concentrations of doxorubicin was presented. The results illustrate that it is possible to gain realistic estimates of the variance of the growth inhibition characteristics, which is of great value in the application of dose-response studies.
Practical considerations
Since the establishment of NCI60, dose-response screens of human tumour cell lines have been one of the most commonly used methods for discovering new anticancer drugs [2]. The approach has mostly been used to discover drugs that are potent in a considerable part of the tested cell lines originating from various tumour types. With this purpose in mind, the bias introduced by analysing cell lines with different doubling times has little or no influence on the conclusions. More recently, the cell line screens have been used to discover treatments that are only potent in a small proportion of the tested cell lines and hence in a small proportion of the cancer patients. Ignoring the doubling time of the cell lines may reduce the capability of discovering such drugs since slowly growing cell lines may appear resistant to the drug.
The issue of growth rate bias may be remedied by using cell lines with approximately the same doubling time or alternatively by conducting the experiments using individual time spans corresponding to a given number of doubling times for each cell line. Based on the latter approach Bracht et al. [19] took the doubling times into account by conducting the dose-response experiments such that each of the 77 cell lines was exposed to the drug for three cell line specific doubling times. For large cell line screens it is neither feasible to generate diverse panels consisting of cancer cell lines with similar doubling times nor is it practical to conduct each experiment for different time spans. The latter option is further complicated since it may not be possible to keep slowly growing cell lines in the exponential growth phase for several doubling times throughout the experiment. The models used in cell line screens NCI60 [3], JFCR39 [5], and CMT100 [2] are based on fixed drug exposure times. We established transformations of these models so that each cell line’s doubling time can be accounted for.
Methodological considerations
Modelling the growth of a cell line exposed to an anticancer drug by the simple differential equation (1) facilitated a meticulous analysis of existing summary statistics for cell line based dose-response studies of growth inhibition. It may be possible to establish a differential equation that leads to either the D or R model. However, the authors have not been able to do so in an unblemished fashion. It is thus difficult to determine which assumptions must be met for the results of these models to be unbiased.
The differential equation was based on exponential cell growth which seems a reasonably assumption since all drug response assays strive toward using the exponential growth phase of the cell lines for the out-read window. Similarly, the rate for cells going into cell cycle arrest or death is assumed exponential and concentration dependent, partly due to computational convenience and partly because no obvious alternative is present. It should be emphasized the assumption of an exponential rate for cells going into cell cycle arrest or death induce a constant drug efficiency throughout the experiment. However, since different drugs induce growth inhibition by different mechanisms, the established differential equation (1) is oversimplified and may therefore model the growth of a cell line exposed to a drug inadequately. It would be interesting to establish more complex systems of differential equations of cell culture growth in combination with more measurements during drug exposure time [20–25]. This would allow estimation of drug induced growth inhibition with improved precision and hence increased biological understanding.
A model-based approach to pre-processing based on a nonlinear regression model was introduced. This model efficiently and simultaneously addresses a number of issues such as background absorbance correction, multiplicative seeding effects and heteroscedastic variance of absorbance measures. All are well-known nuisance effects in cellular/bacterial growth studies [11, 26]. The modelling approach also facilitated outlier detection by residual analysis and standard model checks from regression theory [17]. The dose-response relationship was modelled by the growth curves arising from the solution to the posed differential equation. This lead to pointwise estimates of the dose-response curve of the G-model and interpolation of the curve was done by isotonic regression which is robust against outliers and model misspecifications [14, 27].
Providing precision estimates of the growth inhibition characteristics in this complex setting is not straightforward, so parametric bootstrap of the nonlinear model of the absorbance measurements was used [28]. Alternatively the statistical delta method could have been applied [11, 29]. Although feasible, this would have required complicated approximations by Taylor series expansions, and bootstrapping is generally considered to have superior small sample properties [30].
The dose-response model R (7) based on relative cell counts is very appealing due to its simplicity. Moreover, it is a smooth function so it is possible to fit parametric models to the dose-response curve of R which facilitates extrapolation. When extrapolation is necessary it is possible to fit a parametric model to the dose-response curve of R[31, 32] and subsequently transform the result into the dose-response curve of G using (9). This approach facilitates estimation of time independent summary statistics by extrapolation.
Conclusions
In this study we have shown that conventionally used dose-response models can give rise to biased summary statistics erroneously correlated to the growth rate of the cell lines. We have developed novel summary statistics of dose-response experiments that are applicable on existing data and independent of time under the proposed differential equation. Consequently, we expect that the present approach will be able to improve future drug evaluation studies.
Declarations
Acknowledgements
SF is supported by a Mobility PhD fellowship at the Graduate School of Health, Faculty of Health Sciences, Aarhus University. The research is supported by MSCNET, a translational programme studying cancer stem cells in multiple myeloma supported by the EU FP6, and CHEPRE, a programme studying chemo sensitivity in malignant lymphoma by genomic signatures supported by The Danish Agency for Science, Technology and Innovation, as well as Karen Elise Jensen Fonden. The technical assistance from Ann-Maria Jensen, Louise Hvilshøj Madsen, Helle Høholt, and Helle Stiller is greatly appreciated.
Authors’ Affiliations
References
- Paull KD, Shoemaker RH, Hodes L, Monks A, Scudiero DA, Rubinstein L, Plowman J, Boyd MR: Display and analysis of patterns of differential activity of drugs against human tumor cell lines: development of mean graph and COMPARE algorithm. J Nat Cancer Inst. 1989, 81 (14): 1088-1092. 10.1093/jnci/81.14.1088.View ArticlePubMedGoogle Scholar
- Sharma SV, Haber DA, Settleman J: Cell line-based platforms to evaluate the therapeutic efficacy of candidate anticancer agents. Nat Rev Cancer. 2010, 10 (4): 241-253. 10.1038/nrc2820.View ArticlePubMedGoogle Scholar
- Monks A, Scudiero D, Skehan P: Feasibility of a high-flux anticancer drug screen using a diverse panel of cultured human tumor cell lines. J Nat Cancer Inst. 1991, 83 (11): 757-766. 10.1093/jnci/83.11.757.View ArticlePubMedGoogle Scholar
- Shoemaker RH: The NCI60 human tumour cell line anticancer drug screen. Nat Rev Cancer. 2006, 6 (10): 813-823. 10.1038/nrc1951.View ArticlePubMedGoogle Scholar
- Yamori T, Matsunaga A, Sato S, Yamazaki K, Komi A, Ishizu K, Mita I, Edatsugi H, Matsuba Y, Takezawa K, Nakanishi O, Kohno H, Nakajima Y, Komatsu H, Andoh T, Tsuruo T: Potent antitumor activity of MS-247, a novel DNA minor groove binder, evaluated by an in vitro and in vivo human cancer cell line panel. Cancer Res. 1999, 59 (16): 4042-4049.PubMedGoogle Scholar
- Nakatsu N, Nakamura T, Yamazaki K, Sadahiro S, Makuuchi H, Kanno J, Yamori T: Evaluation of action mechanisms of toxic chemicals using JFCR39, a panel of human cancer cell lines. Mol Pharmacol. 2007, 72 (5): 1171-1180. 10.1124/mol.107.038836.View ArticlePubMedGoogle Scholar
- McDermott U, Sharma SV, Dowell L, Greninger P, Montagut C, Lamb J, Archibald H, Raudales R, Tam A, Lee D, Rothenberg SM, Supko JG, Sordella R, Ulkus LE, Iafrate AJ, Maheswaran S, Njauw CN, Tsao H, Drew L, Hanke JH, Ma XJ, Erlander MG, Gray NS, Haber DA, Settleman J: Identification of genotype-correlated sensitivity to selective kinase inhibitors by using high-throughput tumor cell line profiling. Proc Nat Acad Sci USA. 2007, 104 (50): 19936-19941. 10.1073/pnas.0707498104.View ArticlePubMed CentralPubMedGoogle Scholar
- Azarenko O, Okouneva T, Singletary KW, Jordan MA, Wilson L: Suppression of microtubule dynamic instability and turnover in MCF7 breast cancer cells by sulforaphane. Carcinogenesis. 2008, 29 (12): 2360-2368. 10.1093/carcin/bgn241.View ArticlePubMed CentralPubMedGoogle Scholar
- Liedtke C, Wang J, Tordai A, Symmans WF, Hortobagyi GN, Kiesel L, Hess K, Baggerly KA, Coombes KR, Pusztai L: Clinical evaluation of chemotherapy response predictors developed from breast cancer cell lines. Breast Cancer Res Treat. 2009, 121 (2): 301-309.View ArticlePubMedGoogle Scholar
- Bøgsted M, Holst JM, Fogd K, Falgreen S, Sørensen S, Schmitz A, Bukh A, Johnsen HE, Nyegaard M, Dybkær K: Generation of a predictive melphalan resistance index by drug screen of B-cell cancer cell lines. PloS ONE. 2011, 6 (4): e19322-10.1371/journal.pone.0019322.View ArticleGoogle Scholar
- Boik JC, Narasimhan B: An R package for assessing drug synergism/antagonism. J Stat Softw. 2010, 34 (6): 1-18.View ArticleGoogle Scholar
- Malich G, Markovic B, Winder C: The sensitivity and specificity of the MTS tetrazolium assay for detecting the in vitro cytotoxicity of 20 chemicals using human cell lines. Toxicology. 1997, 124 (3): 179-192. 10.1016/S0300-483X(97)00151-0.View ArticlePubMedGoogle Scholar
- Barlow RE: Statistical Inference Under Order Restrictions: The Theory and Application of Isotonic Regression. 1972, Wiley series in probability and mathematical statistics, Wiley: J. HobokenGoogle Scholar
- Kvam PH, Vidakovic B: Nonparametric Statistics with Applications to Science and Engineering. 2007, Hoboken: WileyView ArticleGoogle Scholar
- Aguirre-Ghiso JA: Models, mechanisms and clinical evidence for cancer dormancy. Nat Rev Cancer. 2007, 7 (11): 834-846. 10.1038/nrc2256.View ArticlePubMed CentralPubMedGoogle Scholar
- Huang S, Pang L: Comparing statistical methods for quantifying drug sensitivity based on in vitro dose-response assays. Assay Drug Dev Technol. 2012, 10: 88-96. 10.1089/adt.2011.0388.View ArticlePubMedGoogle Scholar
- Pinheiro J, Bates DM: Mixed-Effects Models in S and S-PLUS. 2000, New York: Springer VerlagView ArticleGoogle Scholar
- Kondoh E, Mori S, Yamaguchi K, Baba T, Matsumura N, Cory Barnett J, Whitaker RS, Konishi I, Fujii S, Berchuck A, Murphy SK: Targeting slow-proliferating ovarian cancer cells. Int J Cancer. 2010, 126 (10): 2448-2456.PubMedGoogle Scholar
- Bracht K, Nicholls A, Liu Y, Bodmer W: 5-Fluorouracil response in a large panel of colorectal cancer cell lines is associated with mismatch repair deficiency. Br J Cancer. 2010, 103 (3): 340-346. 10.1038/sj.bjc.6605780.View ArticlePubMed CentralPubMedGoogle Scholar
- Gardner SN: A mechanistic, predictive model of dose-response curves for cell cycle phase-specific and -nonspecific drugs. Cancer Res. 2000, 60: 1417-1425.PubMedGoogle Scholar
- Kozusko F, Chen P, Grant SG, Day BW, Panetta JC: A mathematical model of in vitro cancer cell growth and treatment with the antimitotic agent curacin A. Math Biosci. 2001, 170: 1-16. 10.1016/S0025-5564(00)00065-1.View ArticlePubMedGoogle Scholar
- de Pillis LG, Gu W, Radunskaya AE: Mixed immunotherapy and chemotherapy of tumors: modeling, applications and biological interpretations. J Theor Biol. 2006, 238 (4): 841-862. 10.1016/j.jtbi.2005.06.037.View ArticlePubMedGoogle Scholar
- Panetta JC, Evans WE, Cheok MH: Mechanistic mathematical modelling of mercaptopurine effects on cell cycle of human acute lymphoblastic leukaemia cells. Br J Cancer. 2006, 94: 93-100. 10.1038/sj.bjc.6602893.View ArticlePubMed CentralPubMedGoogle Scholar
- Sherer E, Hannemann RE, Rundell A, Ramkrishna D: Analysis of resonance chemotherapy in leukemia treatment via multi-staged population balance models. J Theor Biol. 2006, 240 (4): 648-661. 10.1016/j.jtbi.2005.11.017.View ArticlePubMedGoogle Scholar
- Hamed SS, Roth CM: Mathematical modeling to distinguish cell cycle arrest and cell killing in chemotherapeutic concentration response curves. J Pharmacokinet Pharmacodyn. 2011, 38 (3): 385-403. 10.1007/s10928-011-9199-z.View ArticlePubMedGoogle Scholar
- Cao R, Francisco-Fernández M, Quinto EJ: A random effect multiplicative heteroscedastic model for bacterial growth. BMC Bioinformatics. 2010, 11: 77-10.1186/1471-2105-11-77.View ArticlePubMed CentralPubMedGoogle Scholar
- Lin D, Shkedy Z, Yekutieli D, Amaratunga D, Bijnens L: Modeling Dose-Response Microarray Data in Early Drug Development Experiments Using R. 2012, New York: Springer VerlagView ArticleGoogle Scholar
- Efron B: Bootstrap methods: another look at the jackknife. Ann Stat. 1979, 7: 1-26. 10.1214/aos/1176344552.View ArticleGoogle Scholar
- Van Der Vaart: Asymptotic Statistics, Volume 3 of Cambridge Series in Statistical and Probabilistic Mathematics. 2000, New York: Cambridge University PressGoogle Scholar
- Efron B: The Jackknife, the Bootstrap and other Resampling Plans, Volume 38 of CBMS-NSF Regional Conference Series in Applied Mathematics. 1982, Philadelphia: Society for Industrial and Applied MathematicsView ArticleGoogle Scholar
- Ritz C, Streibig JC: Nonlinear Regression with R. 2008, New York: Springer VerlagGoogle Scholar
- Ritz C, Streibig J: Bioassay analysis using R. J Stat Softw. 2005, 12 (5): 1-22.View ArticleGoogle Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.