entropy and mutual information-[from book of cover]

《entropy and mutual information-[from book of cover]》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

1、CHAPTER2ENTROPY,RELATIVEENTROPY,ANDMUTUALINFORMATIONInthischapterweintroducemostofthebasicdefinitionsrequiredforsubsequentdevelopmentofthetheory.Itisirresistibletoplaywiththeirrelationshipsandinterpretations,takingfaithintheirlaterutility.Afterdefiningentropya

2、ndmutualinformation,weestablishchainrules,thenonnegativityofmutualinformation,thedata-processinginequality,andillustratethesedefinitionsbyexaminingsufficientstatisticsandFano’sinequality.Theconceptofinformationistoobroadtobecapturedcompletelybyasingledefinition

3、.However,foranyprobabilitydistribution,wedefineaquantitycalledtheentropy,whichhasmanypropertiesthatagreewiththeintuitivenotionofwhatameasureofinformationshouldbe.Thisnotionisextendedtodefinemutualinformation,whichisameasureoftheamountofinformationonerandomvari

4、ablecontainsaboutanother.Entropythenbecomestheself-informationofarandomvariable.Mutualinformationisaspecialcaseofamoregeneralquantitycalledrelativeentropy,whichisameasureofthedistancebetweentwoprobabilitydistributions.Allthesequantitiesarecloselyrelatedandsh

5、areanumberofsimpleproperties,someofwhichwederiveinthischapter.Inlaterchaptersweshowhowthesequantitiesariseasnaturalanswerstoanumberofquestionsincommunication,statistics,complexity,andgambling.Thatwillbetheultimatetestofthevalueofthesedefinitions.2.1ENTROPYWefi

6、rstintroducetheconceptofentropy,whichisameasureoftheuncertaintyofarandomvariable.LetXbeadiscreterandomvariablewithalphabetXandprobabilitymassfunctionp(x)=Pr{X=x},x∈X.ElementsofInformationTheory,SecondEdition,ByThomasM.CoverandJoyA.ThomasCopyright2006JohnWil

7、ey&Sons,Inc.1314ENTROPY,RELATIVEENTROPY,ANDMUTUALINFORMATIONWedenotetheprobabilitymassfunctionbyp(x)ratherthanpX(x),forconvenience.Thus,p(x)andp(y)refertotwodifferentrandomvariablesandareinfactdifferentprobabilitymassfunctions,pX(x)andpY(y),respectively.Defin

8、itionTheentropyH(X)ofadiscreterandomvariableXisdefinedby
H(X)=−p(x)logp(x).(2.1)x∈XWealsowriteH(p)fortheabovequantity.Thelogistothebase2andentropyisexpressedinbits.Forexample,theentropyofafaircoi