QConNY2012-NathanMarz-StormDistributedandFaultTolerantRealtimeComputation

《QConNY2012-NathanMarz-StormDistributedandFaultTolerantRealtimeComputation》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、StormDistributedandfault-tolerantrealtimecomputationNathanMarzTwitterBasicinfo•OpensourcedSeptember19th•Implementationis15,000linesofcode•Usedbyover25companies•>3400watchersonGithub(mostwatchedJVMproject)•Veryactivemailinglist•>4000messages•>1000membersBeforeStormQueuesWorkersExample(simplified

2、)ExampleWorkersschemifytweetsandappendtoHadoopExampleWorkersupdatestatisticsonURLsbyincrementingcountersinCassandraExampleUsemod/hashingtomakesuresameURLalwaysgoestosameworkerScalingDeployReconfigure/redeployProblems•Scalingispainful•Poorfault-tolerance•CodingistediousWhatwewant•Guaranteeddatap

3、rocessing•Horizontalscalability•Fault-tolerance•Nointermediatemessagebrokers!•Higherlevelabstractionthanmessagepassing•“Justworks”StormGuaranteeddataprocessingHorizontalscalabilityFault-toleranceNointermediatemessagebrokers!Higherlevelabstractionthanmessagepassing“Justworks”UsecasesStreamDistr

4、ibutedContinuousprocessingRPCcomputationStormClusterStormClusterMasternode(similartoHadoopJobTracker)StormClusterUsedforclustercoordinationStormClusterRunworkerprocessesStartingatopologyKillingatopologyConcepts•Streams•Spouts•Bolts•TopologiesStreamsTupleTupleTupleTupleTupleTupleTupleUnboundeds

5、equenceoftuplesSpoutsSourceofstreamsSpoutexamples•ReadfromKestrelqueue•ReadfromTwitterstreamingAPIBoltsProcessesinputstreamsandproducesnewstreamsBolts•Functions•Filters•Aggregation•Joins•TalktodatabasesTopologyNetworkofspoutsandboltsTasksSpoutsandboltsexecuteasmanytasksacrosstheclusterTaskexec

6、utionTasksarespreadacrosstheclusterTaskexecutionTasksarespreadacrosstheclusterStreamgroupingWhenatupleisemitted,whichtaskdoesitgoto?Streamgrouping•Shufflegrouping:pickarandomtask•Fieldsgrouping:modhashingonasubsetoftuplefields•Allgrouping:sendtoalltasks•Globalgrouping:picktaskwithlowestidTopolog

7、yshuffle[“id1”,“id2”]shuffle[“url”]shuffleallStreamingwordcountTopologyBuilderisusedtoconstructtopologiesinJavaStreamingwordcountDefineaspoutinthetopologywithparallelismof5tasksStreamingwordcountSplitsentencesintowordswithparallelismof8task