Skip to main content
# Table 1 Selective summary of variable selection methods with types of regularizers, main regularization parameters and computational efficiency. Here we focus on the main regularization parameters of the different methods, but there are often several additional hyper-parameters

Method | Regularizer (parameters) | Comments on computational efficiency |
---|---|---|

| ||

Information criteria (4), e.g. AIC [24], BIC [25], EBIC [26] | \(\ell _0\)-penalty (\(\lambda\)) | Best subset selection not efficient for high-dimensional problems. Heuristic optimization [27,28,29] or mixed-integer optimization [30] can be used. |

Lasso [1] | \(\ell _1\)-penalty (\(\lambda\)) | Computationally efficient convex relaxation of \(\ell _0\)-type problem. |

\(\ell _1\)-penalty (\(\lambda , \gamma\)) | Combination of \(\ell _1\)-regularized and unregularized (restricted least squares) estimator. Computationally efficient but tuning more costly than for lasso. | |

Elastic net [4] | \(\ell_1\)-/\(\ell_2\)-penalty (\(\lambda , \alpha\)) | Combination of \(\ell _1\)- and \(\ell _2\)-penalties. Computationally efficient but tuning more costly than for lasso. |

| ||

\(L_2\)Boosting [10] (Algorithm 1) | Early stopping (\(m_{{\text {stop}}}\)) | Tuning of stopping iteration \(m_{{\text {stop}}}\) via resampling leads to implicit regularization. |

Twin boosting [31] | Early stopping (\(m_1,m_2\)) | Two-stage approach using \(L_2\)Boosting estimates as weights in second stage of \(L_2\)Boosting. Tuning more costly than for single-stage \(L_2\)Boosting. |

Flexible (PFER) | Computationally intensive ensemble approach, applying e.g. lasso or \(L_2\)Boosting multiple times on subsamples. Provides control over false positives (PFER). | |

New Subspace Boosting: SubBoost (Algorithm 2), RSubBoost and AdaSubBoost (Algorithm 3) | Automatic stopping (\(\Phi\)) | Multivariable base-learners with |