资源描述：
《基于hadoop的云存储》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。
1、1基于Hadoop的云存储云计算随时随地访问您的应用2云存储的数据管理特点与挑战特点挑战计算资源是可伸缩的数据具有备份数据存储在大量分布的节点之上数据的自我管理和自调优基于大量节点的查询优化算法基于大量节点的索引结构资源调度和负载均衡多租户情况3我们面临的问题您如何来管理大量的应用程序？运行任务来处理100百万兆字节的数据花费11天在一台电脑上读取数据需要大量低价的计算机故障处理速度问题（15分钟修复1000台计算机），但…可靠性问题在大型计算机集群中，每天都有计算机出现故障集群的规模不断变化需要通用
2、的基础架构必须是高效且可靠的4解决方案开源的Apache项目Hadoop主要包括:DistributedFileSystem–分布的数据Map/Reduce–分布的应用程序使用Java开发运行在Linux,MacOS/X,Windows,andSolaris廉价的硬件设备5Typicallyin2levelarchitectureNodesarecommodityPCs40nodes/rackUplinkfromrackis8gigabitRack-internalis1gigabitHardware
3、ClusterofHadoop6DistributedFileSystemSinglenamespaceforentireclusterManagedbyasinglenamenode.Filesaresingle-writerandappend-only.Optimizedforstreamingreadsoflargefiles.Filesarebrokenintolargeblocks.Typically128MBReplicatedtoseveraldatanodes,forreliabili
4、tyAccessfromJava,C,orcommandline.7BlockPlacementDefaultis3replicas,butsettableBlocksareplaced(writesarepipelined):OnsamenodeOndifferentrackOntheotherrackClientsreadfromclosestreplicaIfthereplicationforablockdropsbelowtarget,itisautomaticallyre-replicate
5、d.8HowisYahoousingHadoop?StartedwithbuildingbetterapplicationsScaleupwebscalebatchapplications(search,ads,…)Factoroutcommoncodefromexistingsystems,sonewapplicationswillbeeasiertowriteManagethemanyclusters9RunningProductionWebMapSearchneedsagraphofthe“kn
6、own”webInvertedges,computelinktext,wholegraphheuristicsPeriodicbatchjobusingMap/ReduceUsesachainof~100map/reducejobsScale1trillionedgesingraphLargestshuffleis450TBFinaloutputis300TBcompressedRunson10,000coresRawdiskused5PB10TerabyteSortBenchmarkStartedb
7、yJimGrayatMicrosoftin1998Sorting10billion100byterecordsHadoopwonthegeneralcategoryin209seconds910nodes2quad-coreXeons@2.0Ghz/node4SATAdisks/node8GBram/node1gbethernet/node40nodes/rack8gbethernetuplink/rackPreviousrecordswas297seconds11HadoopclustersWeha
8、ve~20,000machinesrunningHadoopOurlargestclustersarecurrently2000nodesSeveralpetabytesofuserdata(compressed,unreplicated)Werunhundredsofthousandsofjobseverymonth12ResearchClusterUsage13WhoUsesHadoop?Amazon/A9AOLFacebookFoxinteract
