large-scalemachinelearningforbigdata-131029114138-phpapp01

《large-scalemachinelearningforbigdata-131029114138-phpapp01》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、Scalable,Distributed,MachineLearningforBigDataYuHuangSunnyvale,Californiayu.huang07@gmail.comOutlineBigData-Volume,Variety,VelocityParallelComputingandCloudcomputingLambdaarchitecture:Batch,SpeedandServingLayers◦Hadoop:MRimplementationfromYahoo◦Chukwa,

2、Hive,Pig,Zookeeper◦YARN(MRv2ornextgenHadoop)◦Flume,Scribe,Summingbird,DataFreeway◦Storm:StreamprocessingfromTwitter◦S4:StreamprocessingfromYahoo◦Puma:StreamprocessingfromFacebook◦Samza:streamprocessingfromLinkedInNoSQL-NotOnlySQLdatabase◦Bigtable,Dynamo,

3、Cassandra,Hbase.LargeScaleMachineLearning◦Spark–Lightning-FastClusterComputing◦Mahout-ScalableMLonHadoop◦Jubatus–DistributedOnlineReal-timeML◦Graphlab–BigLearningonGraphsatUCBerkeley◦VowpalWabbit–FastLearningatYahoo/MS◦TridentMLandStormPattern:MLonStorm,

4、YARN◦Upcoming---Samoa:MLonS4,StormReferenceBigDataVolume(largeamountsofdatagathered);Variety(variousdegreesofstructure);Velocity(howdataflow,athighrates);Value(business);Variability(changes);Veracity(quality).DataasaService(DaaS)inthecloud;Twomai

5、nstrategiesfordealingwithbigdata:◦Sampling;◦Distributedsystems.BigDataChallenges◦Protectingprivacy;◦Integrationofbigdatatechnologiesintoenterpriselandscape;◦Addressingincreasingrealtimeneedswithincreasingdatavolumeandvarieties;◦Leveragingcloudcomputingwi

6、thbigdatastorageandprocessing.BigDataAnalysisPipelineParallelComputingData-Instruction;◦SIMD,MIMD,…Dataintensive◦CloudcomputingComputeintensive◦GPUcomputingSharedmemory:OpenMPDistributedmemory:MPI,Hybrid:MRCloudComputingAmodelforenablingconvenient,

7、on-demandnetworkaccesstoasharedpoolofconfigurablecomputingresources(Ethernet),usuallyforlargeInternetservices;Dynamicprovisionofservices&resourcepoolsinacoordinatedfashion;Cloudcomputinginfrastructureisjustawebserviceinterfacetooperatingsystemvirtualiza

8、tion(viahypervisor);Heterogeneousbyvirtualization;Everythingasaservice(XaaS);Dataintensive:bigdata;Distributedparallel,morelikeutilitycomputing;Notgridcomputing.X-as-a-ServiceLambdaArchitectureEquation“query=f