The Derivation about Convolutional Neural Networks

《The Derivation about Convolutional Neural Networks》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、TheDerivationaboutConvolutionalNeuralNetworksXuTangNovember5,20141VanillaBack-propagationThroughFullyCon-nectedNetworks1.1FeedforwardPassForamulticlassproblemwithcclassesandNtrainingexamples,thiserrorisgivenby:1∑N∑cEN=(tn−yn)2(1)kk2n=1k=1Becausetheerrorovert

2、hewholedatasetisjustasumovertheindividualerrorsoneachpattern,wewillconsiderbackpropagationwithrespecttoasinglepattern,saythen-thone:∑cn1nn21nn2E=(tk−yk)=∥t−y∥2(2)22k=1Letldenotethecurrentlayer,withtheoutputlayerdesignatedtobelayerLandtheinputlayerdesignatedt

3、obelayer1.Definetheoutputofthislayertobe:xl=f(ul);withul=Wlx(l−1)+bl(3)1.2BackpropagationPassTheerrorswhichwepropagatebackwardsthroughthenetworkcanbethoughtofassensitivitiesofeachunitwithrespecttoperturbationsofthebias.Thatistosay,@E@E@u==(4)@b@u@bsinceinthi

4、scase@u=1.@b1Itisthisderivativethatisbackpropagatedfromhigherlayerstolowerlayers,usingthefollowingrecurrencerelation:l@E=@ul@E@xl=@xl@ul@E′l(5)=◦f(u)@xl∑@E@ul+1=Si◦f′(ul)ll+1@xli=1@ui=(Wl+1)Tl+1◦f′(ul)where“◦”denoteselement-wisemultiplication,andf′(ul)canb

5、edenotedbythis:@E◦f′(ul):::@E◦f′(ul)@xl11@xl1v111v@E◦f′(ul)=......∈Ru∗v(6)@xl...@E◦f′(ul):::@E◦f′(ul)@xlu1@xluv111vThesensitivitiesfortheoutputlayerneuronswilltakeaslightlydifferentform:L@E=@uL@E@xL(7)=@xL@uL=(yn−tn)◦f′(uL)Invectorform,thisiscomputed

6、asanouterproductbetweenthevectorofinputs(whicharetheoutputsfromthepreviouslayer)andthevectorofsensitivities:@E@E@ul=@Wl@ul@Wl@E(@(ul)T)T=@ul@Wl@E@(xl−1)T(Wl)TT(8)=()@ul@(Wl)T@El−1T=(x)@ul=(xl−1)Tll@E∆W=−(9)@Wl22ConvolutionalNeuralNetworks2.1ConvolutionLaye

7、rsEachoutputmapmaycombineconvolutionswithmultipleinputmaps.Ingeneral,wehavethat:∑xl=f(xl−1∗kl+bl)jiijj(10)i∈MjwhereMjrepresentsaselectionofinputmaps,andtheconvolutionisofthe“valid”borderhandlingtypewhenimplementedinMATLAB.2.1.1ComputingtheGradientsWecanrepea

8、tthesamecomputationforeachmapjintheconvolutionallayer,pairingitwiththecorrespondingmapinthesubsamplinglayer:l@Ej=l@uj@E@xl=(11)@xl@ulj@E′l=l◦f(uj)@xjandweknowthat:(@E@E@ujl+1)=@xl@ul+1@xl(12)jj