{"history":[],"article":{"id":"26e78a6a-ec53-4c83-b9bd-de0d4eb382b0","title":"Lasso (statistics)","slug":"lasso-statistics","content":":::summary\n:::\n\n# \nIn [statistics](https://bliptext.com/articles/statistics) and [machine learning](https://bliptext.com/articles/machine-learning), **lasso** (**least absolute shrinkage and selection operator**; also **Lasso** or **LASSO**) is a [regression analysis](https://bliptext.com/articles/regression-analysis) method that performs both [variable selection](https://bliptext.com/articles/variable-selection) and [regularization](https://bliptext.com/articles/regularization-mathematics) in order to enhance the prediction accuracy and interpretability of the resulting [statistical model](https://bliptext.com/articles/statistical-model).\nThe lasso method assumes that the coefficients of the linear model are sparse, meaning that few of them are non-zero.\nIt was originally introduced in [geophysics](https://bliptext.com/articles/geophysics), and later by [Robert Tibshirani](https://bliptext.com/articles/robert-tibshirani), who coined the term.\nLasso was originally formulated for [linear regression](https://bliptext.com/articles/linear-regression) models.\nThis simple case reveals a substantial amount about the estimator.\nThese include its relationship to [ridge regression](https://bliptext.com/articles/ridge-regression) and [best subset selection](https://bliptext.com/articles/best-subset-selection) and the connections between lasso coefficient estimates and so-called soft thresholding.\nIt also reveals that (like standard linear regression) the coefficient estimates do not need to be unique if [covariates](https://bliptext.com/articles/covariate) are [collinear](https://bliptext.com/articles/collinear).\nThough originally defined for linear regression, lasso regularization is easily extended to other statistical models including [generalized linear models](https://bliptext.com/articles/generalized-linear-model), [generalized estimating equations](https://bliptext.com/articles/generalized-estimating-equation), [proportional hazards models](https://bliptext.com/articles/proportional-hazards-model), and [M-estimators](https://bliptext.com/articles/m-estimator).\nLasso's ability to perform subset selection relies on the form of the constraint and has a variety of interpretations including in terms of [geometry](https://bliptext.com/articles/geometry), [Bayesian statistics](https://bliptext.com/articles/bayesian-statistics) and [convex analysis](https://bliptext.com/articles/convex-analysis).\nThe LASSO is closely related to [basis pursuit denoising](https://bliptext.com/articles/basis-pursuit-denoising).\n\n# History\nLasso was introduced in order to improve the prediction accuracy and interpretability of regression models.\nIt selects a reduced set of the known covariates for use in a model.\nLasso was developed independently in geophysics literature in 1986, based on prior work that used the [\\ell^1](https://bliptext.com/articles/l1-norm) [penalty](https://bliptext.com/articles/penalized-least-squares) for both fitting and penalization of the coefficients.\nStatistician [Robert Tibshirani](https://bliptext.com/articles/robert-tibshirani) independently rediscovered and popularized it in 1996, based on [Breiman](https://bliptext.com/articles/leo-breiman)'s nonnegative garrote.\nPrior to lasso, the most widely used method for choosing covariates was [stepwise selection](https://bliptext.com/articles/stepwise-regression).\nThat approach only improves prediction accuracy in certain cases, such as when only a few covariates have a strong relationship with the outcome.\nHowever, in other cases, it can increase prediction error.\nAt the time, [ridge regression](https://bliptext.com/articles/tikhonov-regularization) was the most popular technique for improving prediction accuracy.\nRidge regression improves prediction error by [shrinking](https://bliptext.com/articles/shrinkage-statistics) the sum of the squares of the [regression coefficients](https://bliptext.com/articles/regression-coefficients) to be less than a fixed value in order to reduce [overfitting](https://bliptext.com/articles/overfitting), but it does not perform covariate selection and therefore does not help to make the model more interpretable.\nLasso achieves both of these goals by forcing the sum of the absolute value of the regression coefficients to be less than a fixed value, which forces certain coefficients to zero, excluding them from impacting prediction.\nThis idea is similar to ridge regression, which also shrinks the size of the coefficients; however, ridge regression does not set coefficients to zero (and, thus, does not perform [variable selection](https://bliptext.com/articles/variable-selection)).\n\n# Basic form\n\n## Least squares\nConsider a sample consisting of N cases, each of which consists of p [covariates](https://bliptext.com/articles/covariate) and a single outcome.\nLet y_i be the outcome and be the covariate vector for the i th case.\nThen the objective of lasso is to solve\nHere \\beta_0 is the constant coefficient, is the coefficient vector, and t is a prespecified free parameter that determines the degree of regularization.\nLetting X be the covariate matrix, so that and x_i^T is the i th row of X, the expression can be written more compactly as\nwhere is the standard [\\ell^p norm](https://bliptext.com/articles/lp-space).\nDenoting the scalar mean of the data points x_i by \\bar{x} and the mean of the response variables y_i by \\bar{y}, the resulting estimate for \\beta_0 is, so that\nand therefore it is standard to work with variables that have been made zero-mean.\nAdditionally, the covariates are typically [standardized](https://bliptext.com/articles/standardized-statistics) so that the solution does not depend on the measurement scale.\nIt can be helpful to rewrite\nin the so-called [Lagrangian](https://bliptext.com/articles/lagrange-multiplier) form\nwhere the exact relationship between t and \\lambda is data dependent.\n\n## Orthonormal covariates\nSome basic properties of the lasso estimator can now be considered.\nAssuming first that the covariates are [orthonormal](https://bliptext.com/articles/orthonormal) so that, where \\delta_{ij} is the [Kronecker delta](https://bliptext.com/articles/kronecker-delta), or, equivalently, X^T X = I, then using [subgradient methods](https://bliptext.com/articles/subgradient-methods) it can be shown that\nS_\\alpha is referred to as the soft thresholding operator, since it translates values towards zero (making them exactly zero if they are small enough) instead of setting smaller values to zero and leaving larger ones untouched as the hard thresholding operator, often denoted H_\\alpha, would.\nIn ridge regression the objective is to minimize\nUsing X^TX = I and the ridge regression formula:, it yields:\nRidge regression shrinks all coefficients by a uniform factor of and does not set any coefficients to zero.\nIt can also be compared to regression with [best subset selection](https://bliptext.com/articles/best-subset-selection), in which the goal is to minimize\nwhere is the \"\\ell^0 norm\", which is defined as \\| z \\| = m if exactly m components of z are nonzero.\nIn this case, it can be shown that\nwhere H_\\alpha is the so-called hard thresholding function and \\mathrm{I} is an indicator function (it is 1 if its argument is true and 0 otherwise).\nTherefore, the lasso estimates share features of both ridge and best subset selection regression since they both shrink the magnitude of all the coefficients, like ridge regression and set some of them to zero, as in the best subset selection case.\nAdditionally, while ridge regression scales all of the coefficients by a constant factor, lasso instead translates the coefficients towards zero by a constant value and sets them to zero if they reach it.\n\n## Correlated covariates\nIn one special case two covariates, say j and k, are identical for each observation, so that, where.\nThen the values of \\beta_j and \\beta_k that minimize the lasso objective function are not uniquely determined.\nIn fact, if some \\hat{\\beta} in which, then if s \\in [0,1] replacing by and by , while keeping all the other fixed, gives a new solution, so the lasso objective function then has a continuum of valid minimizers.\nSeveral variants of the lasso, including the [Elastic net regularization](https://bliptext.com/articles/elastic-net-regularization), have been designed to address this shortcoming.\n\n# General form\nLasso regularization can be extended to other objective functions such as those for [generalized linear models](https://bliptext.com/articles/generalized-linear-model), [generalized estimating equations](https://bliptext.com/articles/generalized-estimating-equation), [proportional hazards models](https://bliptext.com/articles/proportional-hazards-model), and [M-estimators](https://bliptext.com/articles/m-estimator).\nGiven the objective function\nthe lasso regularized version of the estimator s the solution to\nwhere only \\beta is penalized while \\alpha is free to take any allowed value, just as \\beta_0 was not penalized in the basic case.\n\n# Interpretations\n\n## Geometric interpretation\nLasso can set coefficients to zero, while the superficially similar ridge regression cannot.\nThis is due to the difference in the shape of their constraint boundaries.\nBoth lasso and ridge regression can be interpreted as minimizing the same objective function\nbut with respect to different constraints: for lasso and for ridge.\nThe figure shows that the constraint region defined by the \\ell^1 norm is a square rotated so that its corners lie on the axes (in general a [cross-polytope](https://bliptext.com/articles/cross-polytope)), while the region defined by the \\ell^2 norm is a circle (in general an n-sphere), which is [rotationally](https://bliptext.com/articles/rotation-mathematics) [invariant](https://bliptext.com/articles/invariant-mathematics) and, therefore, has no corners.\nAs seen in the figure, a convex object that lies tangent to the boundary, such as the line shown, is likely to encounter a corner (or a higher-dimensional equivalent) of a hypercube, for which some components of \\beta are identically zero, while in the case of an n-sphere, the points on the boundary for which some of the components of \\beta are zero are not distinguished from the others and the convex object is no more likely to contact a point at which some components of \\beta are zero than one for which none of them are.\n# [Forms of the constraint regions for lasso and ridge regression. | upload.wikimedia.org/wikipedia/commons/f/f8/L1///and///L2///balls.svg]\n\n## Making λ easier to interpret with an accuracy-simplicity tradeoff\nThe lasso can be rescaled so that it becomes easy to anticipate and influence the degree of shrinkage associated with a given value of \\lambda.\nIt is assumed that X is standardized with z-scores and that y is centered (zero mean).\nLet \\beta_0 represent the hypothesized regression coefficients and let refer to the data-optimized ordinary least squares solutions.\nWe can then define the [Lagrangian](https://bliptext.com/articles/lagrange-multiplier) as a tradeoff between the in-sample accuracy of the data-optimized solutions and the simplicity of sticking to the hypothesized values.\nThis results in\nwhere q_i is specified below and the \"prime\" symbol stands for transpose.\nThe first fraction represents relative accuracy, the second fraction relative simplicity, and \\lambda balances between the two.\nGiven a single regressor, relative simplicity can be defined by specifying q_i as, which is the maximum amount of deviation from \\beta_0 when \\lambda=0.\nAssuming that \\beta_{0}=0, the solution path can be defined in terms of [R^2](https://bliptext.com/articles/coefficient-of-determination):\nIf \\lambda=0, the ordinary least squares solution (OLS) is used.\nThe hypothesized value of \\beta_0=0 is selected if \\lambda is bigger than R^2.\nFurthermore, if R^2=1, then \\lambda represents the proportional influence of \\beta_0=0.\nIn other words, measures in percentage terms the minimal amount of influence of the hypothesized value relative to the data-optimized OLS solution.\nIf an \\ell_2-norm is used to penalize deviations from zero given a single regressor, the solution path is given by\nLike b_{\\ell_1}, b_{\\ell_2} moves in the direction of the point when \\lambda is close to zero; but unlike b_{\\ell_1}, the influence of R^2 diminishes in b_{\\ell_2} if \\lambda increases (see figure).\nGiven multiple regressors, the moment that a parameter is activated (i.e. allowed to deviate from \\beta_0) is also determined by a regressor's contribution to R^2 accuracy.\nFirst,\nAn R^2 of 75% means that in-sample accuracy improves by 75% if the unrestricted OLS solutions are used instead of the hypothesized \\beta_0 values.\nThe individual contribution of deviating from each hypothesis can be computed with the p x p matrix\nwhere.\nIf when R^2 is computed, then the diagonal elements of R^{\\otimes} sum to R^2.\nThe diagonal R^{\\otimes} values may be smaller than 0 or, less often, larger than 1.\nIf regressors are uncorrelated, then the i^{th} diagonal element of R^{\\otimes} simply corresponds to the r^2 value between x_i and y.\nA rescaled version of the adaptive lasso of can be obtained by setting.\nIf regressors are uncorrelated, the moment that the i^{th} parameter is activated is given by the i^{th} diagonal element of R^{\\otimes}.\nAssuming for convenience that \\beta_0 is a vector of zeros,\nThat is, if regressors are uncorrelated, \\lambda again specifies the minimal influence of \\beta_0.\nEven when regressors are correlated, the first time that a regression parameter is activated occurs when \\lambda is equal to the highest diagonal element of R^{\\otimes}.\nThese results can be compared to a rescaled version of the lasso by defining, which is the average absolute deviation of from \\beta_0.\nAssuming that regressors are uncorrelated, then the moment of activation of the i^{th} regressor is given by\nFor p=1, the moment of activation is again given by.\nIf \\beta_0 is a vector of zeros and a subset of p_B relevant parameters are equally responsible for a perfect fit of R^2=1, then this subset is activated at a \\lambda value of \\frac{1}{p}.\nThe moment of activation of a relevant regressor then equals.\nIn other words, the inclusion of irrelevant regressors delays the moment that relevant regressors are activated by this rescaled lasso.\nThe adaptive lasso and the lasso are special cases of a '1ASTc' estimator.\nThe latter only groups parameters together if the absolute correlation among regressors is larger than a user-specified value.\n# [Solution paths for the \\ell_1 norm and \\ell_2 norm when and \\beta_{0}=0 | upload.wikimedia.org/wikipedia/commons/7/7b/SolutionPathL1L2.svg]\n\n## Bayesian interpretation\nJust as ridge regression can be interpreted as linear regression for which the coefficients have been assigned normal [prior distributions](https://bliptext.com/articles/prior-distribution), lasso can be interpreted as linear regression for which the coefficients have Laplace [prior distributions](https://bliptext.com/articles/prior-distribution).\nThe Laplace distribution is sharply [peaked](https://bliptext.com/articles/peakedness) at zero (its first derivative is discontinuous at zero) and it concentrates its probability mass closer to zero than does the normal distribution.\nThis provides an alternative explanation of why lasso tends to set some coefficients to zero, while ridge regression does not.\n# [Laplace distributions are sharply peaked at their mean with more probability density concentrated there compared to a normal distribution. | upload.wikimedia.org/wikipedia/commons/0/0a/Laplace///pdf///mod.svg]\n\n## Convex relaxation interpretation\nLasso can also be viewed as a convex relaxation of the best subset selection regression problem, which is to find the subset of \\leq k covariates that results in the smallest value of the objective function for some fixed k \\leq n, where n is the total number of covariates.\nThe \"\\ell^0 norm\",, (the number of nonzero entries of a vector), is the limiting case of \"\\ell^p norms\", of the form (where the quotation marks signify that these are not really norms for p < 1 since is not convex for p < 1, so the triangle inequality does not hold).\nTherefore, since p = 1 is the smallest value for which the \"\\ell^p norm\" is convex (and therefore actually a norm), lasso is, in some sense, the best convex approximation to the best subset selection problem, since the region defined by is the [convex hull](https://bliptext.com/articles/convex-hull) of the region defined by for p < 1.\n\n# Generalizations\nLasso variants have been created in order to remedy limitations of the original technique and to make the method more useful for particular problems.\nAlmost all of these focus on respecting or exploiting dependencies among the covariates.\n[Elastic net regularization](https://bliptext.com/articles/elastic-net-regularization) adds an additional ridge regression-like penalty that improves performance when the number of predictors is larger than the sample size, allows the method to select strongly correlated variables together, and improves overall prediction accuracy.\nGroup lasso allows groups of related covariates to be selected as a single unit, which can be useful in settings where it does not make sense to include some covariates without others.\nFurther extensions of group lasso perform variable selection within individual groups (sparse group lasso) and allow overlap between groups (overlap group lasso).\nFused lasso can account for the spatial or temporal characteristics of a problem, resulting in estimates that better match system structure.\nLasso-regularized models can be fit using techniques including [subgradient methods](https://bliptext.com/articles/subgradient-methods), [least-angle regression](https://bliptext.com/articles/least-angle-regression) (LARS), and [proximal gradient methods](https://bliptext.com/articles/proximal-gradient-methods).\nDetermining the optimal value for the regularization parameter is an important part of ensuring that the model performs well; it is typically chosen using [cross-validation](https://bliptext.com/articles/cross-validation-statistics).\n\n## Elastic net\nIn 2005, Zou and Hastie introduced the [elastic net](https://bliptext.com/articles/elastic-net-regularization).\nWhen p > n (the number of covariates is greater than the sample size) lasso can select only n covariates (even when more are associated with the outcome) and it tends to select one covariate from any set of highly correlated covariates.\nAdditionally, even when n > p, ridge regression tends to perform better given strongly correlated covariates.\nThe elastic net extends lasso by adding an additional \\ell^2 penalty term giving\nwhich is equivalent to solving\nThis problem can be written in a simple lasso form\nletting\nThen, which, when the covariates are orthogonal to each other, gives\nSo the result of the elastic net penalty is a combination of the effects of the lasso and ridge penalties.\nReturning to the general case, the fact that the penalty function is now strictly convex means that if, , which is a change from lasso.\nIn general, if\nis the sample correlation matrix because the x 's are normalized.\nTherefore, highly correlated covariates tend to have similar regression coefficients, with the degree of similarity depending on both \\| y \\|_1 and \\lambda_2, which is different from lasso.\nThis phenomenon, in which strongly correlated covariates have similar regression coefficients, is referred to as the grouping effect.\nGrouping is desirable since, in applications such as tying genes to a disease, finding all the associated covariates is preferable, rather than selecting one from each set of correlated covariates, as lasso often does.\nIn addition, selecting only one from each group typically results in increased prediction error, since the model is less robust (which is why ridge regression often outperforms lasso).\n\n## Group lasso\nIn 2006, Yuan and Lin introduced the group lasso to allow predefined groups of covariates to jointly be selected into or out of a model.\nThis is useful in many settings, perhaps most obviously when a categorical variable is coded as a collection of binary covariates.\nIn this case, group lasso can ensure that all the variables encoding the categorical covariate are included or excluded together.\nAnother setting in which grouping is natural is in biological studies.\nSince genes and proteins often lie in known pathways, which pathways are related to an outcome may be more significant than whether individual genes are.\nThe objective function for the group lasso is a natural generalization of the standard lasso objective\nwhere the [design matrix](https://bliptext.com/articles/design-matrix) X and covariate vector \\beta have been replaced by a collection of design matrices X_j and covariate vectors \\beta_j, one for each of the J groups.\nAdditionally, the penalty term is now a sum over \\ell^2 norms defined by the positive definite matrices K_j.\nIf each covariate is in its own group and K_j = I, then this reduces to the standard lasso, while if there is only a single group and K_1 = I, it reduces to ridge regression.\nSince the penalty reduces to an \\ell^2 norm on the subspaces defined by each group, it cannot select out only some of the covariates from a group, just as ridge regression cannot.\nHowever, because the penalty is the sum over the different subspace norms, as in the standard lasso, the constraint has some non-differential points, which correspond to some subspaces being identically zero.\nTherefore, it can set the coefficient vectors corresponding to some subspaces to zero, while only shrinking others.\nHowever, it is possible to extend the group lasso to the so-called sparse group lasso, which can select individual covariates within a group, by adding an additional \\ell^1 penalty to each group subspace.\nAnother extension, group lasso with overlap allows covariates to be shared across groups, e.g., if a gene were to occur in two pathways.\nThe \"gglasso\" package by in R, allows for fast and efficient implementation of Group LASSO.\n\n## Fused lasso\nIn some cases, the phenomenon under study may have important spatial or temporal structure that must be considered during analysis, such as time series or image-based data.\nIn 2005, Tibshirani and colleagues introduced the fused lasso to extend the use of lasso to this type of data.\nThe fused lasso objective function is\nThe first constraint is the lasso constraint, while the second directly penalizes large changes with respect to the temporal or spatial structure, which forces the coefficients to vary smoothly to reflect the system's underlying logic.\nClustered lasso is a generalization of fused lasso that identifies and groups relevant covariates based on their effects (coefficients).\nThe basic idea is to penalize the differences between the coefficients so that nonzero ones cluster.\nThis can be modeled using the following regularization:\nIn contrast, variables can be clustered into highly correlated groups, and then a single representative covariate can be extracted from each cluster.\nAlgorithms exist that solve the fused lasso problem, and some generalizations of it.\nAlgorithms can solve it exactly in a finite number of operations.\n\n## Quasi-norms and bridge regression\nLasso, elastic net, group and fused lasso construct the penalty functions from the \\ell^1 and \\ell^2 norms (with weights, if necessary).\nThe bridge regression utilises general \\ell^p norms (p \\geq 1) and quasinorms (0< p < 1).\nFor example, for p=1/2 the analogue of lasso objective in the Lagrangian form is to solve\nwhere\nIt is claimed that the fractional quasi-norms \\ell^p (0< p < 1) provide more meaningful results in data analysis both theoretically and empirically.\nThe non-convexity of these quasi-norms complicates the optimization problem.\nTo solve this problem, an expectation-minimization procedure is developed and implemented for minimization of function\nwhere is an arbitrary concave monotonically increasing function (for example, gives the lasso penalty and gives the \\ell^{1/2} penalty).\nThe efficient algorithm for minimization is based on piece-wise [quadratic approximation](https://bliptext.com/articles/taylor-s-theorem) of subquadratic growth (PQSQ).\n# [An example of a PQSQ (piece-wise quadratic function of subquadratic growth) potential function u(x); here the majorant function is f(x)=x; the potential is defined with trimming after r_3. | upload.wikimedia.org/wikipedia/commons/4/45/PQSQ///potential.png]\n# [An example how efficient PQSQ regularized regression works just as \\ell^1-norm lasso. | upload.wikimedia.org/wikipedia/commons/4/40/PQSQ2.png]\n\n## Adaptive lasso\nThe adaptive lasso was introduced by Zou in 2006 for linear regression and by Zhang and Lu in 2007 for proportional hazards regression.\n\n## Prior lasso\nThe prior lasso was introduced for generalized linear models by Jiang et al. in 2016 to incorporate prior information, such as the importance of certain covariates.\nIn prior lasso, such information is summarized into pseudo responses (called prior responses) and then an additional criterion function is added to the usual objective function with a lasso penalty.\nWithout loss of generality, in linear regression, the new objective function can be written as\nwhich is equivalent to\nthe usual lasso objective function with the responses y being replaced by a weighted average of the observed responses and the prior responses (called the adjusted response values by the prior information).\nIn prior lasso, the parameter \\eta is called a balancing parameter, in that it balances the relative importance of the data and the prior information.\nIn the extreme case of \\eta = 0, prior lasso is reduced to lasso.\nIf, prior lasso will solely rely on the prior information to fit the model.\nFurthermore, the balancing parameter \\eta has another appealing interpretation: it controls the variance of \\beta in its prior distribution from a Bayesian viewpoint.\nPrior lasso is more efficient in parameter estimation and prediction (with a smaller estimation error and prediction error) when the prior information is of high quality, and is robust to the low quality prior information with a good choice of the balancing parameter \\eta.\n\n# Computing lasso solutions\nThe loss function of the lasso is not differentiable, but a wide variety of techniques from convex analysis and optimization theory have been developed to compute the solutions path of the lasso.\nThese include coordinate descent, subgradient methods, [least-angle regression](https://bliptext.com/articles/least-angle-regression) (LARS), and proximal gradient methods.\n[Subgradient](https://bliptext.com/articles/subderivative) methods are the natural generalization of traditional methods such as [gradient descent](https://bliptext.com/articles/gradient-descent) and stochastic [gradient descent](https://bliptext.com/articles/gradient-descent) to the case in which the objective function is not differentiable at all points.\nLARS is a method that is closely tied to lasso models, and in many cases allows them to be fit efficiently, though it may not perform well in all circumstances.\nLARS generates complete solution paths.\nProximal methods have become popular because of their flexibility and performance and are an area of active research.\nThe choice of method will depend on the particular lasso variant, the data and the available resources.\nHowever, proximal methods generally perform well.\nThe \"glmnet\" package in R, where \"glm\" is a reference to \"generalized linear models\" and \"net\" refers to the \"net\" from \"elastic net\" provides an extremely efficient way to implement LASSO and some of its variants.\nThe \"celer\" package in Python provides a highly efficient solver for the Lasso problem, often outperforming traditional solvers like scikit-learn by up to 100 times in certain scenarios, particularly with high-dimensional datasets.\nThis package leverages dual extrapolation techniques to achieve its performance gains.\nThe celer package is available at [GitHub](https://bliptext.com/articles/undefined).\n\n# Choice of regularization parameter\nChoosing the regularization parameter (\\lambda) is a fundamental part of lasso.\nA good value is essential to the performance of lasso since it controls the strength of shrinkage and variable selection, which, in moderation can improve both prediction accuracy and interpretability.\nHowever, if the regularization becomes too strong, important variables may be omitted and coefficients may be shrunk excessively, which can harm both predictive capacity and inferencing.\n[Cross-validation](https://bliptext.com/articles/cross-validation-statistics) is often used to find the regularization parameter.\nInformation criteria such as the [Bayesian information criterion](https://bliptext.com/articles/bayesian-information-criterion) (BIC) and the [Akaike information criterion](https://bliptext.com/articles/akaike-information-criterion) (AIC) might be preferable to cross-validation, because they are faster to compute and their performance is less volatile in small samples.\nAn information criterion selects the estimator's regularization parameter by maximizing a model's in-sample accuracy while penalizing its effective number of parameters/degrees of freedom.\nZou et al. proposed to measure the effective degrees of freedom by counting the number of parameters that deviate from zero.\nThe degrees of freedom approach was considered flawed by Kaufman and Rosset and Janson et al., because a model's degrees of freedom might increase even when it is penalized harder by the regularization parameter.\nAs an alternative, the relative simplicity measure defined above can be used to count the effective number of parameters.\nFor the lasso, this measure is given by\nwhich monotonically increases from zero to p as the regularization parameter decreases from \\infty to zero.\n\n# Selected applications\nLASSO has been applied in economics and finance, and was found to improve prediction and to select sometimes neglected variables, for example in corporate bankruptcy prediction literature, or high growth firms prediction.","current_revision":null,"created_by":"kACnuQmH5k9Nr5VNVEpVeg7XpvUmMctD","created_at":"2025-01-26T11:37:52.796Z","updated_at":"2025-01-26T11:37:52.796Z","search_vector":"'lasso':1 'statist':2","search_content":" \n \n\n# \nIn [statistics](https://bliptext.com/articles/statistics) and [machine learning](https://bliptext.com/articles/machine-learning), **lasso** (**least absolute shrinkage and selection operator**; also **Lasso** or **LASSO**) is a [regression analysis](https://bliptext.com/articles/regression-analysis) method that performs both [variable selection](https://bliptext.com/articles/variable-selection) and [regularization](https://bliptext.com/articles/regularization-mathematics) in order to enhance the prediction accuracy and interpretability of the resulting [statistical model](https://bliptext.com/articles/statistical-model).\nThe lasso method assumes that the coefficients of the linear model are sparse, meaning that few of them are non-zero.\nIt was originally introduced in [geophysics](https://bliptext.com/articles/geophysics), and later by [Robert Tibshirani](https://bliptext.com/articles/robert-tibshirani), who coined the term.\nLasso was originally formulated for [linear regression](https://bliptext.com/articles/linear-regression) models.\nThis simple case reveals a substantial amount about the estimator.\nThese include its relationship to [ridge regression](https://bliptext.com/articles/ridge-regression) and [best subset selection](https://bliptext.com/articles/best-subset-selection) and the connections between lasso coefficient estimates and so-called soft thresholding.\nIt also reveals that (like standard linear regression) the coefficient estimates do not need to be unique if [covariates](https://bliptext.com/articles/covariate) are [collinear](https://bliptext.com/articles/collinear).\nThough originally defined for linear regression, lasso regularization is easily extended to other statistical models including [generalized linear models](https://bliptext.com/articles/generalized-linear-model), [generalized estimating equations](https://bliptext.com/articles/generalized-estimating-equation), [proportional hazards models](https://bliptext.com/articles/proportional-hazards-model), and [M-estimators](https://bliptext.com/articles/m-estimator).\nLasso's ability to perform subset selection relies on the form of the constraint and has a variety of interpretations including in terms of [geometry](https://bliptext.com/articles/geometry), [Bayesian statistics](https://bliptext.com/articles/bayesian-statistics) and [convex analysis](https://bliptext.com/articles/convex-analysis).\nThe LASSO is closely related to [basis pursuit denoising](https://bliptext.com/articles/basis-pursuit-denoising).\n\n# History\nLasso was introduced in order to improve the prediction accuracy and interpretability of regression models.\nIt selects a reduced set of the known covariates for use in a model.\nLasso was developed independently in geophysics literature in 1986, based on prior work that used the [\\ell^1](https://bliptext.com/articles/l1-norm) [penalty](https://bliptext.com/articles/penalized-least-squares) for both fitting and penalization of the coefficients.\nStatistician [Robert Tibshirani](https://bliptext.com/articles/robert-tibshirani) independently rediscovered and popularized it in 1996, based on [Breiman](https://bliptext.com/articles/leo-breiman)'s nonnegative garrote.\nPrior to lasso, the most widely used method for choosing covariates was [stepwise selection](https://bliptext.com/articles/stepwise-regression).\nThat approach only improves prediction accuracy in certain cases, such as when only a few covariates have a strong relationship with the outcome.\nHowever, in other cases, it can increase prediction error.\nAt the time, [ridge regression](https://bliptext.com/articles/tikhonov-regularization) was the most popular technique for improving prediction accuracy.\nRidge regression improves prediction error by [shrinking](https://bliptext.com/articles/shrinkage-statistics) the sum of the squares of the [regression coefficients](https://bliptext.com/articles/regression-coefficients) to be less than a fixed value in order to reduce [overfitting](https://bliptext.com/articles/overfitting), but it does not perform covariate selection and therefore does not help to make the model more interpretable.\nLasso achieves both of these goals by forcing the sum of the absolute value of the regression coefficients to be less than a fixed value, which forces certain coefficients to zero, excluding them from impacting prediction.\nThis idea is similar to ridge regression, which also shrinks the size of the coefficients; however, ridge regression does not set coefficients to zero (and, thus, does not perform [variable selection](https://bliptext.com/articles/variable-selection)).\n\n# Basic form\n\n## Least squares\nConsider a sample consisting of N cases, each of which consists of p [covariates](https://bliptext.com/articles/covariate) and a single outcome.\nLet y_i be the outcome and be the covariate vector for the i th case.\nThen the objective of lasso is to solve\nHere \\beta_0 is the constant coefficient, is the coefficient vector, and t is a prespecified free parameter that determines the degree of regularization.\nLetting X be the covariate matrix, so that and x_i^T is the i th row of X, the expression can be written more compactly as\nwhere is the standard [\\ell^p norm](https://bliptext.com/articles/lp-space).\nDenoting the scalar mean of the data points x_i by \\bar{x} and the mean of the response variables y_i by \\bar{y}, the resulting estimate for \\beta_0 is, so that\nand therefore it is standard to work with variables that have been made zero-mean.\nAdditionally, the covariates are typically [standardized](https://bliptext.com/articles/standardized-statistics) so that the solution does not depend on the measurement scale.\nIt can be helpful to rewrite\nin the so-called [Lagrangian](https://bliptext.com/articles/lagrange-multiplier) form\nwhere the exact relationship between t and \\lambda is data dependent.\n\n## Orthonormal covariates\nSome basic properties of the lasso estimator can now be considered.\nAssuming first that the covariates are [orthonormal](https://bliptext.com/articles/orthonormal) so that, where \\delta_{ij} is the [Kronecker delta](https://bliptext.com/articles/kronecker-delta), or, equivalently, X^T X = I, then using [subgradient methods](https://bliptext.com/articles/subgradient-methods) it can be shown that\nS_\\alpha is referred to as the soft thresholding operator, since it translates values towards zero (making them exactly zero if they are small enough) instead of setting smaller values to zero and leaving larger ones untouched as the hard thresholding operator, often denoted H_\\alpha, would.\nIn ridge regression the objective is to minimize\nUsing X^TX = I and the ridge regression formula:, it yields:\nRidge regression shrinks all coefficients by a uniform factor of and does not set any coefficients to zero.\nIt can also be compared to regression with [best subset selection](https://bliptext.com/articles/best-subset-selection), in which the goal is to minimize\nwhere is the \"\\ell^0 norm\", which is defined as \\| z \\| = m if exactly m components of z are nonzero.\nIn this case, it can be shown that\nwhere H_\\alpha is the so-called hard thresholding function and \\mathrm{I} is an indicator function (it is 1 if its argument is true and 0 otherwise).\nTherefore, the lasso estimates share features of both ridge and best subset selection regression since they both shrink the magnitude of all the coefficients, like ridge regression and set some of them to zero, as in the best subset selection case.\nAdditionally, while ridge regression scales all of the coefficients by a constant factor, lasso instead translates the coefficients towards zero by a constant value and sets them to zero if they reach it.\n\n## Correlated covariates\nIn one special case two covariates, say j and k, are identical for each observation, so that, where.\nThen the values of \\beta_j and \\beta_k that minimize the lasso objective function are not uniquely determined.\nIn fact, if some \\hat{\\beta} in which, then if s \\in [0,1] replacing by and by , while keeping all the other fixed, gives a new solution, so the lasso objective function then has a continuum of valid minimizers.\nSeveral variants of the lasso, including the [Elastic net regularization](https://bliptext.com/articles/elastic-net-regularization), have been designed to address this shortcoming.\n\n# General form\nLasso regularization can be extended to other objective functions such as those for [generalized linear models](https://bliptext.com/articles/generalized-linear-model), [generalized estimating equations](https://bliptext.com/articles/generalized-estimating-equation), [proportional hazards models](https://bliptext.com/articles/proportional-hazards-model), and [M-estimators](https://bliptext.com/articles/m-estimator).\nGiven the objective function\nthe lasso regularized version of the estimator s the solution to\nwhere only \\beta is penalized while \\alpha is free to take any allowed value, just as \\beta_0 was not penalized in the basic case.\n\n# Interpretations\n\n## Geometric interpretation\nLasso can set coefficients to zero, while the superficially similar ridge regression cannot.\nThis is due to the difference in the shape of their constraint boundaries.\nBoth lasso and ridge regression can be interpreted as minimizing the same objective function\nbut with respect to different constraints: for lasso and for ridge.\nThe figure shows that the constraint region defined by the \\ell^1 norm is a square rotated so that its corners lie on the axes (in general a [cross-polytope](https://bliptext.com/articles/cross-polytope)), while the region defined by the \\ell^2 norm is a circle (in general an n-sphere), which is [rotationally](https://bliptext.com/articles/rotation-mathematics) [invariant](https://bliptext.com/articles/invariant-mathematics) and, therefore, has no corners.\nAs seen in the figure, a convex object that lies tangent to the boundary, such as the line shown, is likely to encounter a corner (or a higher-dimensional equivalent) of a hypercube, for which some components of \\beta are identically zero, while in the case of an n-sphere, the points on the boundary for which some of the components of \\beta are zero are not distinguished from the others and the convex object is no more likely to contact a point at which some components of \\beta are zero than one for which none of them are.\n# [Forms of the constraint regions for lasso and ridge regression. | upload.wikimedia.org/wikipedia/commons/f/f8/L1///and///L2///balls.svg]\n\n## Making λ easier to interpret with an accuracy-simplicity tradeoff\nThe lasso can be rescaled so that it becomes easy to anticipate and influence the degree of shrinkage associated with a given value of \\lambda.\nIt is assumed that X is standardized with z-scores and that y is centered (zero mean).\nLet \\beta_0 represent the hypothesized regression coefficients and let refer to the data-optimized ordinary least squares solutions.\nWe can then define the [Lagrangian](https://bliptext.com/articles/lagrange-multiplier) as a tradeoff between the in-sample accuracy of the data-optimized solutions and the simplicity of sticking to the hypothesized values.\nThis results in\nwhere q_i is specified below and the \"prime\" symbol stands for transpose.\nThe first fraction represents relative accuracy, the second fraction relative simplicity, and \\lambda balances between the two.\nGiven a single regressor, relative simplicity can be defined by specifying q_i as, which is the maximum amount of deviation from \\beta_0 when \\lambda=0.\nAssuming that \\beta_{0}=0, the solution path can be defined in terms of [R^2](https://bliptext.com/articles/coefficient-of-determination):\nIf \\lambda=0, the ordinary least squares solution (OLS) is used.\nThe hypothesized value of \\beta_0=0 is selected if \\lambda is bigger than R^2.\nFurthermore, if R^2=1, then \\lambda represents the proportional influence of \\beta_0=0.\nIn other words, measures in percentage terms the minimal amount of influence of the hypothesized value relative to the data-optimized OLS solution.\nIf an \\ell_2-norm is used to penalize deviations from zero given a single regressor, the solution path is given by\nLike b_{\\ell_1}, b_{\\ell_2} moves in the direction of the point when \\lambda is close to zero; but unlike b_{\\ell_1}, the influence of R^2 diminishes in b_{\\ell_2} if \\lambda increases (see figure).\nGiven multiple regressors, the moment that a parameter is activated (i.e. allowed to deviate from \\beta_0) is also determined by a regressor's contribution to R^2 accuracy.\nFirst,\nAn R^2 of 75% means that in-sample accuracy improves by 75% if the unrestricted OLS solutions are used instead of the hypothesized \\beta_0 values.\nThe individual contribution of deviating from each hypothesis can be computed with the p x p matrix\nwhere.\nIf when R^2 is computed, then the diagonal elements of R^{\\otimes} sum to R^2.\nThe diagonal R^{\\otimes} values may be smaller than 0 or, less often, larger than 1.\nIf regressors are uncorrelated, then the i^{th} diagonal element of R^{\\otimes} simply corresponds to the r^2 value between x_i and y.\nA rescaled version of the adaptive lasso of can be obtained by setting.\nIf regressors are uncorrelated, the moment that the i^{th} parameter is activated is given by the i^{th} diagonal element of R^{\\otimes}.\nAssuming for convenience that \\beta_0 is a vector of zeros,\nThat is, if regressors are uncorrelated, \\lambda again specifies the minimal influence of \\beta_0.\nEven when regressors are correlated, the first time that a regression parameter is activated occurs when \\lambda is equal to the highest diagonal element of R^{\\otimes}.\nThese results can be compared to a rescaled version of the lasso by defining, which is the average absolute deviation of from \\beta_0.\nAssuming that regressors are uncorrelated, then the moment of activation of the i^{th} regressor is given by\nFor p=1, the moment of activation is again given by.\nIf \\beta_0 is a vector of zeros and a subset of p_B relevant parameters are equally responsible for a perfect fit of R^2=1, then this subset is activated at a \\lambda value of \\frac{1}{p}.\nThe moment of activation of a relevant regressor then equals.\nIn other words, the inclusion of irrelevant regressors delays the moment that relevant regressors are activated by this rescaled lasso.\nThe adaptive lasso and the lasso are special cases of a '1ASTc' estimator.\nThe latter only groups parameters together if the absolute correlation among regressors is larger than a user-specified value.\n# [Solution paths for the \\ell_1 norm and \\ell_2 norm when and \\beta_{0}=0 | upload.wikimedia.org/wikipedia/commons/7/7b/SolutionPathL1L2.svg]\n\n## Bayesian interpretation\nJust as ridge regression can be interpreted as linear regression for which the coefficients have been assigned normal [prior distributions](https://bliptext.com/articles/prior-distribution), lasso can be interpreted as linear regression for which the coefficients have Laplace [prior distributions](https://bliptext.com/articles/prior-distribution).\nThe Laplace distribution is sharply [peaked](https://bliptext.com/articles/peakedness) at zero (its first derivative is discontinuous at zero) and it concentrates its probability mass closer to zero than does the normal distribution.\nThis provides an alternative explanation of why lasso tends to set some coefficients to zero, while ridge regression does not.\n# [Laplace distributions are sharply peaked at their mean with more probability density concentrated there compared to a normal distribution. | upload.wikimedia.org/wikipedia/commons/0/0a/Laplace///pdf///mod.svg]\n\n## Convex relaxation interpretation\nLasso can also be viewed as a convex relaxation of the best subset selection regression problem, which is to find the subset of \\leq k covariates that results in the smallest value of the objective function for some fixed k \\leq n, where n is the total number of covariates.\nThe \"\\ell^0 norm\",, (the number of nonzero entries of a vector), is the limiting case of \"\\ell^p norms\", of the form (where the quotation marks signify that these are not really norms for p n (the number of covariates is greater than the sample size) lasso can select only n covariates (even when more are associated with the outcome) and it tends to select one covariate from any set of highly correlated covariates.\nAdditionally, even when n > p, ridge regression tends to perform better given strongly correlated covariates.\nThe elastic net extends lasso by adding an additional \\ell^2 penalty term giving\nwhich is equivalent to solving\nThis problem can be written in a simple lasso form\nletting\nThen, which, when the covariates are orthogonal to each other, gives\nSo the result of the elastic net penalty is a combination of the effects of the lasso and ridge penalties.\nReturning to the general case, the fact that the penalty function is now strictly convex means that if, , which is a change from lasso.\nIn general, if\nis the sample correlation matrix because the x 's are normalized.\nTherefore, highly correlated covariates tend to have similar regression coefficients, with the degree of similarity depending on both \\| y \\|_1 and \\lambda_2, which is different from lasso.\nThis phenomenon, in which strongly correlated covariates have similar regression coefficients, is referred to as the grouping effect.\nGrouping is desirable since, in applications such as tying genes to a disease, finding all the associated covariates is preferable, rather than selecting one from each set of correlated covariates, as lasso often does.\nIn addition, selecting only one from each group typically results in increased prediction error, since the model is less robust (which is why ridge regression often outperforms lasso).\n\n## Group lasso\nIn 2006, Yuan and Lin introduced the group lasso to allow predefined groups of covariates to jointly be selected into or out of a model.\nThis is useful in many settings, perhaps most obviously when a categorical variable is coded as a collection of binary covariates.\nIn this case, group lasso can ensure that all the variables encoding the categorical covariate are included or excluded together.\nAnother setting in which grouping is natural is in biological studies.\nSince genes and proteins often lie in known pathways, which pathways are related to an outcome may be more significant than whether individual genes are.\nThe objective function for the group lasso is a natural generalization of the standard lasso objective\nwhere the [design matrix](https://bliptext.com/articles/design-matrix) X and covariate vector \\beta have been replaced by a collection of design matrices X_j and covariate vectors \\beta_j, one for each of the J groups.\nAdditionally, the penalty term is now a sum over \\ell^2 norms defined by the positive definite matrices K_j.\nIf each covariate is in its own group and K_j = I, then this reduces to the standard lasso, while if there is only a single group and K_1 = I, it reduces to ridge regression.\nSince the penalty reduces to an \\ell^2 norm on the subspaces defined by each group, it cannot select out only some of the covariates from a group, just as ridge regression cannot.\nHowever, because the penalty is the sum over the different subspace norms, as in the standard lasso, the constraint has some non-differential points, which correspond to some subspaces being identically zero.\nTherefore, it can set the coefficient vectors corresponding to some subspaces to zero, while only shrinking others.\nHowever, it is possible to extend the group lasso to the so-called sparse group lasso, which can select individual covariates within a group, by adding an additional \\ell^1 penalty to each group subspace.\nAnother extension, group lasso with overlap allows covariates to be shared across groups, e.g., if a gene were to occur in two pathways.\nThe \"gglasso\" package by in R, allows for fast and efficient implementation of Group LASSO.\n\n## Fused lasso\nIn some cases, the phenomenon under study may have important spatial or temporal structure that must be considered during analysis, such as time series or image-based data.\nIn 2005, Tibshirani and colleagues introduced the fused lasso to extend the use of lasso to this type of data.\nThe fused lasso objective function is\nThe first constraint is the lasso constraint, while the second directly penalizes large changes with respect to the temporal or spatial structure, which forces the coefficients to vary smoothly to reflect the system's underlying logic.\nClustered lasso is a generalization of fused lasso that identifies and groups relevant covariates based on their effects (coefficients).\nThe basic idea is to penalize the differences between the coefficients so that nonzero ones cluster.\nThis can be modeled using the following regularization:\nIn contrast, variables can be clustered into highly correlated groups, and then a single representative covariate can be extracted from each cluster.\nAlgorithms exist that solve the fused lasso problem, and some generalizations of it.\nAlgorithms can solve it exactly in a finite number of operations.\n\n## Quasi-norms and bridge regression\nLasso, elastic net, group and fused lasso construct the penalty functions from the \\ell^1 and \\ell^2 norms (with weights, if necessary).\nThe bridge regression utilises general \\ell^p norms (p \\geq 1) and quasinorms (0< p < 1).\nFor example, for p=1/2 the analogue of lasso objective in the Lagrangian form is to solve\nwhere\nIt is claimed that the fractional quasi-norms \\ell^p (0< p < 1) provide more meaningful results in data analysis both theoretically and empirically.\nThe non-convexity of these quasi-norms complicates the optimization problem.\nTo solve this problem, an expectation-minimization procedure is developed and implemented for minimization of function\nwhere is an arbitrary concave monotonically increasing function (for example, gives the lasso penalty and gives the \\ell^{1/2} penalty).\nThe efficient algorithm for minimization is based on piece-wise [quadratic approximation](https://bliptext.com/articles/taylor-s-theorem) of subquadratic growth (PQSQ).\n# [An example of a PQSQ (piece-wise quadratic function of subquadratic growth) potential function u(x); here the majorant function is f(x)=x; the potential is defined with trimming after r_3. | upload.wikimedia.org/wikipedia/commons/4/45/PQSQ///potential.png]\n# [An example how efficient PQSQ regularized regression works just as \\ell^1-norm lasso. | upload.wikimedia.org/wikipedia/commons/4/40/PQSQ2.png]\n\n## Adaptive lasso\nThe adaptive lasso was introduced by Zou in 2006 for linear regression and by Zhang and Lu in 2007 for proportional hazards regression.\n\n## Prior lasso\nThe prior lasso was introduced for generalized linear models by Jiang et al. in 2016 to incorporate prior information, such as the importance of certain covariates.\nIn prior lasso, such information is summarized into pseudo responses (called prior responses) and then an additional criterion function is added to the usual objective function with a lasso penalty.\nWithout loss of generality, in linear regression, the new objective function can be written as\nwhich is equivalent to\nthe usual lasso objective function with the responses y being replaced by a weighted average of the observed responses and the prior responses (called the adjusted response values by the prior information).\nIn prior lasso, the parameter \\eta is called a balancing parameter, in that it balances the relative importance of the data and the prior information.\nIn the extreme case of \\eta = 0, prior lasso is reduced to lasso.\nIf, prior lasso will solely rely on the prior information to fit the model.\nFurthermore, the balancing parameter \\eta has another appealing interpretation: it controls the variance of \\beta in its prior distribution from a Bayesian viewpoint.\nPrior lasso is more efficient in parameter estimation and prediction (with a smaller estimation error and prediction error) when the prior information is of high quality, and is robust to the low quality prior information with a good choice of the balancing parameter \\eta.\n\n# Computing lasso solutions\nThe loss function of the lasso is not differentiable, but a wide variety of techniques from convex analysis and optimization theory have been developed to compute the solutions path of the lasso.\nThese include coordinate descent, subgradient methods, [least-angle regression](https://bliptext.com/articles/least-angle-regression) (LARS), and proximal gradient methods.\n[Subgradient](https://bliptext.com/articles/subderivative) methods are the natural generalization of traditional methods such as [gradient descent](https://bliptext.com/articles/gradient-descent) and stochastic [gradient descent](https://bliptext.com/articles/gradient-descent) to the case in which the objective function is not differentiable at all points.\nLARS is a method that is closely tied to lasso models, and in many cases allows them to be fit efficiently, though it may not perform well in all circumstances.\nLARS generates complete solution paths.\nProximal methods have become popular because of their flexibility and performance and are an area of active research.\nThe choice of method will depend on the particular lasso variant, the data and the available resources.\nHowever, proximal methods generally perform well.\nThe \"glmnet\" package in R, where \"glm\" is a reference to \"generalized linear models\" and \"net\" refers to the \"net\" from \"elastic net\" provides an extremely efficient way to implement LASSO and some of its variants.\nThe \"celer\" package in Python provides a highly efficient solver for the Lasso problem, often outperforming traditional solvers like scikit-learn by up to 100 times in certain scenarios, particularly with high-dimensional datasets.\nThis package leverages dual extrapolation techniques to achieve its performance gains.\nThe celer package is available at [GitHub](https://bliptext.com/articles/undefined).\n\n# Choice of regularization parameter\nChoosing the regularization parameter (\\lambda) is a fundamental part of lasso.\nA good value is essential to the performance of lasso since it controls the strength of shrinkage and variable selection, which, in moderation can improve both prediction accuracy and interpretability.\nHowever, if the regularization becomes too strong, important variables may be omitted and coefficients may be shrunk excessively, which can harm both predictive capacity and inferencing.\n[Cross-validation](https://bliptext.com/articles/cross-validation-statistics) is often used to find the regularization parameter.\nInformation criteria such as the [Bayesian information criterion](https://bliptext.com/articles/bayesian-information-criterion) (BIC) and the [Akaike information criterion](https://bliptext.com/articles/akaike-information-criterion) (AIC) might be preferable to cross-validation, because they are faster to compute and their performance is less volatile in small samples.\nAn information criterion selects the estimator's regularization parameter by maximizing a model's in-sample accuracy while penalizing its effective number of parameters/degrees of freedom.\nZou et al. proposed to measure the effective degrees of freedom by counting the number of parameters that deviate from zero.\nThe degrees of freedom approach was considered flawed by Kaufman and Rosset and Janson et al., because a model's degrees of freedom might increase even when it is penalized harder by the regularization parameter.\nAs an alternative, the relative simplicity measure defined above can be used to count the effective number of parameters.\nFor the lasso, this measure is given by\nwhich monotonically increases from zero to p as the regularization parameter decreases from \\infty to zero.\n\n# Selected applications\nLASSO has been applied in economics and finance, and was found to improve prediction and to select sometimes neglected variables, for example in corporate bankruptcy prediction literature, or high growth firms prediction.","revisionCount":0},"hasMore":false,"nextCursor":null}