Michael Stonebraker

《Michael Stonebraker》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、WhatDoes‘BigData’MeanandWhoWillWin?MichaelStonebrakerTheMeaningofBigData-3V’s•BigVolume—Businessstuffwithsimple(SQL)analytics—Businessstuffwithcomplex(non-SQL)analytics•BigVelocity—Drinkfromafirehose•BigVariety—Largenumberofdiversedatasourcestointegrate2BigVolume-LittleAnalytics•Well

2、addressedbydatawarehousecrowd•WhoareprettygoodatSQLanalyticson—Hundredsofnodes—Petabytesofdata•Buttherehasbeenadramaticseachangeinthelastdecade!3TraditionalRowStoreArchitecture(circa2005)Record1Record2Record3Record4E.g.DB2,Oracle,Sybase,SQLServer,Greenplum,Netezza,Teradata,…ColumnSto

3、reArchitecture(now)CCCOOOLLLUUUMMMNNN12KE.g.Vertica,SybaseIQ,MicrosoftPDW,DB2-BLU,RedShift,…TheFacts•Acolumnstoreis50Xfasterthanarowstore!!!!•Thesuccessfulvendorswillallrun“sharenothing”multi-nodecolumnstoresforscalability•Mosteverybody–exceptOracle–hasgottenthereoverthelastdecadeOra

4、cleExadataisnot:acolumnstoreascalableshared-nothingarchitecture6BigData-BigAnalyticsTheNextBigThing!•Complexmathoperations(machinelearning,clustering,trenddetection,….)—Theworldofthe“quants”andthe“rocketscientists”—Mostlyspecifiedaslinearalgebraonarraydata•Adozenorsocommon‘innerloops

5、’—Matrixmultiply—QRdecomposition—SVDdecomposition—Linearregression7BigData-BigAnalyticsAnExample•Considerclosingpriceonalltradingdaysforthelast5yearsfortwostocksAandB•Whatisthecovariancebetweenthetwotime-series?(1/N)*sum(Ai-mean(A))*(Bi-mean(B))8NowMakeItInteresting…•Dothisforallpair

6、sof4000stocks—Thedataisthefollowing4000x1000matrixStockttttttt….t12345671000S1S2…S40009ArrayAnswer•Ignoringthe(1/N)andsubtractingoffthemeans….Stock*StockT•TrythisinSQLwithsomerelationalsimulationofthestockarray!!!!10DataScientistJobDescription•Cleanandcuratethedata(majorityofhistimer

7、ightnow–forwardpointertoVariety)•Until(tired){Datamanagementoperation(s);Complexanalyticsoperations(s);}11UseCaseRequirements•Complexanalytics—Covarianceisjustthestart—Definedonarrays—Graphsarejustsparsearrays!!!•Datamanagement—Leaveoutoutliers—Justonsecuritieswithamarketcapover$10B•

8、Scalabilityt