Druid_ Interactive Queries Meet Real-time Data Presentation.pdf

《Druid_ Interactive Queries Meet Real-time Data Presentation.pdf》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、DRUID:INTERACTIVEQUERIESMEETREAL-TIMEDATAERICTSCHETTERDEMO“REQUIREMENTS”REQUIREMENTS•DataIngestionRate•Ingestdataandmakeitqueryableinreal-time•ArbitraryDrill-Downs,Slice-n-Dice•Arbitrarybooleanfilters•Availability•Downtimeisevil“WHATWETRIED”WHATWETRIEDI.RD

2、BMS-RelationalDatabaseI.RDBMS-THESETUP•StarSchema•AggregateTables•QueryCachingI.RDBMS-THERESULTS•Queriesthatwerecached•fast•Queriesagainstaggregatetables•fasttoacceptable•Queriesagainstbasefacttable•generallyunacceptableI.RDBMS-PERFORMANCESelectCOUNT(*)sc

3、anrate~5.5Mrows/second/core1dayofsummarizedaggregates60M+rows1queryover1week,16cores~5secondsPageloadwith20queriesoveraweeklongtimeofdataWHATWETRIEDI.RDBMS-RelationalDatabaseWHATWETRIEDI.RDBMS-RelationalDatabaseII.NoSQL-Key/ValueStoreII.NOSQL-THESETUP•Pre

4、-aggregatealldimensionalcombinations(truncatetime)•StoreresultsinaNoSQLstoreKeyValue1revenue=$1.19tsgenderagerevenue1,Mrevenue=$0.151M18$0.151,Frevenue=$1.041F25$1.031,18revenue=$0.162F18$0.011,25revenue=$1.031,M,18revenue=$0.151,F,18revenue=$0.011,F,25re

5、venue=$1.03II.NOSQL-THERESULTS•Querieswerefast•rangescanonprimarykey•Inflexible•notaggregated,notavailable•Notcontinuouslyupdated•aggregatefirst,thendisplay•ProcessingscalesexponentiallyII.NOSQL-PERFORMANCE•Dimensionalcombinations=>exponentialincrease•Tried

6、limitingdimensionaldepth•stillexpandsexponentially•Example:~500krecords•11dimensions,5-deep•4.5hoursona15-nodeHadoopcluster•14dimensions,5-deep•9hoursona25-nodeHadoopclusterWHATWETRIEDI.RDBMS-RelationalDatabaseII.NoSQL-Key/ValueStoreWHATWETRIEDI.RDBMS-Rel

7、ationalDatabaseII.NoSQL-Key/ValueStoreIII.???WHATWELEARNED•ProblemwithRDBMS:scansareslow•ProblemwithNoSQL:computationallyintractableWHATWELEARNED•ProblemwithRDBMS:scansareslow•ProblemwithNoSQL:computationallyintractable!•TacklingRDBMSissueseemseasier“INTR

8、ODUCINGDRUID”DRUID–KEYFEATURES1.Real-TimeIngestion(Indigestion?)2.Slicing-n-DicingDrillDownFruitNinjas3.AvailableARCHITECTUREARCHITECTUREARCHITECTURERealtimeNodesQueryAPIARCHITECTUREHandOffDataaHistoricalNodesRealti