(Hinton1995)The wake-sleep algorithm for unsupervised neural networks.pdf

《(Hinton1995)The wake-sleep algorithm for unsupervised neural networks.pdf》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、Thewake-sleepalgorithmforunsupervisedneuralnetworksGeoffreyEHintonPeterDayanBrendanJFreyRadfordMNealDepartmentofComputerScienceUniversityofToronto6KingsCollegeRoadTorontoM5S1A4,Canada3rdApril1995AbstractAnunsupervisedlearningalgorithmforamultilayernetworkofstochasticneuronsisdescribed.Bottom-upre

2、cognitionconnectionsconverttheinputintorepresentationsinsuccessivehiddenlayersandtop-downgenerativeconnectionsreconstructtherepresentationinonelayerfromtherepresentationinthelayerabove.Inthewakephase,neuronsaredrivenbyrecognitionconnections,andgenerativeconnectionsareadaptedtoincreasetheprobabilit

3、ythattheywouldreconstructthecorrectactivityvectorinthelayerbelow.Inthesleepphase,neuronsaredrivenbygenerativeconnectionsandrecognitionconnectionsareadaptedtoincreasetheprobabilitythattheywouldproducethecorrectactivityvectorinthelayerabove.Supervisedlearningalgorithmsformultilayerneuralnetworksface

4、twoproblems:Theyrequireateachertospecifythedesiredoutputofthenetworkandtheyrequiresomemethodofcommunicatingerrorinformationtoalloftheconnections.Thewake-sleepalgorithmavoidsboththeseproblems.Whenthereisnoexternalteachingsignaltobematched,someothergoalisrequiredtoforcethehiddenunitstoextractunderly

5、ingstructure.Inthewake-sleepalgorithmthegoalistolearnrepresentationsthatareeconomicaltodescribebutallowtheinputtobereconstructedaccurately.Wecanquantifythisgoalbyimaginingacommunicationgameinwhicheachvectorofrawsensoryinputsiscommunicatedtoareceiverbyfirstsendingitshiddenrepresentationandthensendin

6、gthedifferencebetweentheinputvectoranditstop-downreconstructionfromthehiddenrepresentation.Theaimoflearningistominimizethedescriptionlengthwhichisthetotalnumberofbitsthatwouldberequiredtocommunicatetheinputvectorsinthisway[1].Nocommunicationactuallytakesplace,butminimizingthedescriptionlengththatw

7、ouldberequiredforcesthenetworktolearneconomicalrepresentationsthatcapturetheunderlyingregularitiesinthedata[2].1Theneuralnetworkhastwoquitedifferentsetsofconnections.Thebottom-uprecognitionconnectionsareusedtocon