欢迎来到天天文库
浏览记录
ID:36282483
大小:4.72 MB
页数:42页
时间:2019-05-08
《斯坦福深度学习课件7 Understanding_and_improving_deep_learing_with_random_matrix_theory》由会员上传分享,免费在线阅读,更多相关内容在学术论文-天天文库。
1、UnderstandingandImprovingDeepLearningwithRandomMatrixTheoryJeffreyPenningtonGoogleBrain,NYCNovember8,2017Stats385,StanfordConfidential&ProprietaryOutline1.Motivation2.Essentialsofrandommatrixtheory3.Geometryofneuralnetworklosslandscapes4.Resurrectingthesigmoidindeeplearning5.Nonli
2、nearrandommatrixtheory6.ConclusionsConfidential&ProprietaryMotivation:WhyRandomMatrices?Confidential&ProprietaryWhyrandommatrices?●Theinitialweightconfigurationisrandom○Trainingmayinduceonlylow-rankperturbationsaroundtherandomconfiguration●Anexacttheoryofdeeplearningislikelytobein
3、tractableoruninformative○Largecomplexsystemsareoftenwell-modeledwithrandomvariables■E.g.statisticalphysicsandthermodynamics●Manyimportantquantitiesarespecifictomatrixstructure○E.g.eigenvaluesandeigenvectorsConfidential&ProprietaryWhichmatricesdowecareabout?●Activations●Hessians●Ja
4、cobiansConfidential&ProprietaryEssentialsofrandommatrixtheoryConfidential&ProprietarySpectraldensityForanymatrix,theempiricalspectraldensityis:Forasequenceofmatriceswithincreasingsize,,thelimitingspectraldensityis:StieltjestransformFortheStieltjestransformisdefinedas:Usingtheident
5、ities,ThespectraldensitycanberecoveredfromGusingtheinversionformula,R-transformandS-transformTheStieltjestransformcanbeusedtodefinetwousefulauxiliaryobjects:theR-transform,definedbythefunctionalequation,andtheS-transform,definedbyasimilarfunctionalequation,Freeadditionandfreemulti
6、plicationIfAandBarefreelyindependent,thenthespectrumofthesumA+BcomputedusingtheR-transform:AndthespectrumoftheproductABcanbecomputedusingtheS-transform:Confidential&ProprietaryFreeIndependenceClassicalindependenceFreeindependenceareindependentifonehasarefreelyindependentifwhenever
7、andaresuchthatwheneverandaresuchthatConfidential&ProprietaryGeometryofneuralnetworklosssurfaceswithYasamanBahriConfidential&ProprietarySinglecriticalpointConfidential&ProprietaryMultiplecriticalpointsA)AllminimaareroughlyB)GlobalminimummuchC)Allminimaandindex1equivalent,butindex1l
8、owerthanlocalminimacriticalpoints
此文档下载收益归作者所有