Ridge Regression

《Ridge Regression》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、RidgeregressionSelectionofRidgeregressioninR/SASRidgeRegressionPatrickBrehenySeptember1PatrickBrehenyBST764:AppliedStatisticalModeling1/22RidgeregressionDenitionandsolutionSelectionofPropertiesRidgeregressioninR/SASRidgeregression:DenitionAsmentionedinthepreviouslectu

2、re,ridgeregressionpenalizesthesizeoftheregressioncoecientsSpecically,theridgeregressionestimatebisdenedasthevalueofthatminimizesXXp(yxT)2+2iijij=1PatrickBrehenyBST764:AppliedStatisticalModeling2/22RidgeregressionDenitionandsolutionSelectionofPropertiesRidgeregressi

3、oninR/SASRidgeregression:SolutionTheorem:Thesolutiontotheridgeregressionproblemisgivenbyb=(XTX+I)1XTyNotethesimilaritytotheordinaryleastsquaressolution,butwiththeadditionofaridge"downthediagonalridgeOLSCorollary:As!0,b!bridgeCorollary:As!1,b!0PatrickBrehenyBST764:App

4、liedStatisticalModeling3/22RidgeregressionDenitionandsolutionSelectionofPropertiesRidgeregressioninR/SASRidgeregression:Solution(Cont'd)Corollary:Inthespecialcaseofanorthonormaldesignmatrix,^OLS^ridge=JJ1+Thisillustratestheessentialfeatureofridgeregression:shrinkageApp

5、lyingtheridgeregressionpenaltyhastheeectofshrinkingtheestimatestowardzero{introducingbiasbutreducingthevarianceoftheestimatePatrickBrehenyBST764:AppliedStatisticalModeling4/22RidgeregressionDenitionandsolutionSelectionofPropertiesRidgeregressioninR/SASRidgevs.OLSinthep

6、resenceofcollinearityThebenetsofridgeregressionaremoststrikinginthepresenceofmulticollinearity,asillustratedinthefollowingexample:>x1<-rnorm(20)>x2<-rnorm(20,mean=x1,sd=.01)>y<-rnorm(20,mean=3+x1+x2)>lm(y~x1+x2)$coef(Intercept)x1x22.58206439.971344-38.040040>lm.ridge(y~x

7、1+x2,lambda=1)x1x22.62149980.99067730.8973912PatrickBrehenyBST764:AppliedStatisticalModeling5/22RidgeregressionDenitionandsolutionSelectionofPropertiesRidgeregressioninR/SASInvertibilityRecallfromBST760thattheordinaryleastsquaresestimatesdonotalwaysexist;ifXisnotfullran

8、k,XTXisnotOLSinvertibleandthereisnouniquesolutionforbThisproblemdoesnotoccurwithridgeregression,