资源描述：
《基于hadoop的网络数据分析》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。
1、Hadoop-basedInternetTrafficAnalysis2012-03-14YoungseokLeelee@cnu.ac.krChungnamNationalUniversity1Gartner:10KeyITTrendsfor20122DataExplosionDigitaluniverse2005:130ExaByte2timepertwoyears90%ofthedataintheworldtodaywascreatedwithinthepasttwoyearsMotivationChea
2、pstorageMobilecomputing3BigDataBigdatadatasetswhosesizeisbeyondtheabilityofcommonlyusedsoftwaretoolstocapture,manage,andprocessthedatawithinatolerableelapsedtimeExampleWalmart:1millioncustomertransactionseveryhour2.5PBofdataFacebook:40billionphotosfromusers
3、2.5PBofuserdata+15TB/day(4/2009)eBayhas6.5PBofuserdata+50TB/day(5/2009)4PredictingFluTrendswithQueriesGinsberg,J.,etal.(2009).Detectinginfluenzaepidemicsusingsearchenginequerydata.Nature,457,1012-1014.5CuttingEdgeofDistributed/ParallelComputingDistributed:M
4、apReduce1PBsortingbyGoogle2008:6hoursand2minuteson4,000computers2011:33minuteson8000computersParallel:multi-coreCPU,GPU,FPGA40GbpsIPforwardingcapabilityS.Hanetal.,“PacketShader:AGPU-AcceleratedSoftwareRouter“ACMSIGCOMM,20106Open-sourceframeworkforrunningapp
5、licationsonlargeclustersbuiltofcommodityhardwareImplementationofMapReduceandHDFSMapReduce:computationalparadigmHDFS:distributedfilesystem7MapReduceMapReduceisaprogrammingmodelforlargedatasetFirstsuggestedbyGoogleJ.DeanandS.Ghemawat,“MapReduce:SimplifiedData
6、ProcessingonLargeCluster,”OSDI,2004[8]UseronlyspecifyamapandareducefunctionDistributedgrepCountofURLaccessfrequencyReverseweb-linkgraphTerm-vectorperhostInvertedindexDistributedsort8DatafromInternetMeasurementWebwebdoc,blog,twitterIPpacketsraw,header,dnsFlo
7、wCiscoNetFlowv5/v9LogServer,switch,router,sensor,machine,appliancesManagementdataSNMPMIBRoutingtableBGPupdate,tableActiveprobingping,traceroute,skitter9HowLarge?TypePackets4.6TBfor1hrin10GENetFlow1.2GBfor1dayforarouterinCNURoutingtableRouteviewsinbzip2RIBS:
8、45MBevery2hrs->540MBfor1day->197GBfor1yrUpdates:1MBforevery15minutes10Gbps10Gbps10Gbps4.5TB10CAIDADataDatasourcesMacroscopictopologydata(IPv4/IPv6)byArchipelagoTraffictracesatInternetcorebackbonelinksT
