055 Spark Shuffle Introduction

《055 Spark Shuffle Introduction》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、SPARKSHUFFLEINTRODUCTION天火@蘑菇街AboutMeSpark/Hadoop/Hbase/PhoenixcontributorForsparkmainlycontributesin:•Yarn•Shuffle•BlockManager•Scala2.10update•StandaloneHA•Variousotherfixes.tianhuo@mogujie.comWeibo@冷冻蚂蚁blog.csdn.net/colorantWhySparkisfast(er)•Whomdowecompareto?•Whatdowemeanbyfast?•fasttowrite

2、•fasttorunWhySparkisfast(er)cont.•Butthefigureinpreviouspageissomehowmisleading.•Thekeyistheflexibleprogrammingmode.•Whichleadtomorereasonabledataflow.•WhichleadtolessIOoperation.•EspeciallyforiterativeheavyworkloadslikeML.•Whichpotentiallycutoffalotofshuffleoperationsneeded.•But,youwon’talwaysb

3、elucky.•Manyapplogicdidneedtoexchangealotofdata.•Intheend,youwillstillneedtodealwithshuffle•Andwhichusuallyimpactperformancealot.Whatisshuffle7ShuffleoverviewAggregatorAggregatorAggregatorAggregatorAggregatorAggregatorHowdoesshufflecomeintothepicture•Sparkrunjobstagebystage.•StagesarebuildupbyDA

4、GScheduleraccordingtoRDD’sShuffleDependency•e.g.ShuffleRDD/CoGroupedRDDwillhaveaShuffleDependency•ManyoperatorwillcreateShuffleRDD/CoGroupedRDDunderthehook.•Repartition/CombineByKey/GroupBy/ReduceByKey/cogroup•manyotheroperatorwillfurthercallintotheaboveoperators•e.g.variousjoinoperatorwillcallc

5、ogroup.•EachShuffleDependencymapstoonestageinSparkJobandthenwillleadtoashuffle.SoeveryoneshouldhaveseenthisbeforeA: B: G: Stage 1 groupByC: D: F: map E: join Stage 2 union Stage 3 whyshuffleisexpensive•Whendoingshuffle,datanolongerstayinmemoryonly•Forspark,shuffleprocessmightinvolve•datapartitio

6、n:whichmightinvolveveryexpensivedatasortingworksetc.•dataser/deser:toenabledatabeentransferthroughnetworkoracrossprocesses.•datacompression:toreduceIObandwidthetc.•DISKIO:probablymultipletimesononesingledatablock•E.g.ShuffleSpill,MergecombineHistory•Spark0.6-0.7,samecodepathwithRDD’spersistentme

7、thod,canchooseMEMORY_ONLYandDISK_ONLY(default).•Spark0.8-0.9:•separateshufflecodepathfromBMandcreateShuffleBlockManagerandBlockObjectWriteronlyforshuffle,nowshuffledatacanonlybewrittentodisk.•Shuffleoptimization:Consolidates