etl过程优化与增量数据抽取的研究

《etl过程优化与增量数据抽取的研究》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、工程硕士学位论文AbstractAsthekeycomponentofdatawarehouse，ETL(ExtractionTransformationLoading)livestheentirelifecycleofiIatawarehouseandtakesonresponsibilityofintegratingdata．ThisthesisdoesresearchandoptimizationworksontwoaspectsofETL．Ontheonehand，thedataqualityofETLdirectlyaf．fectstheanalysisofupperdecis

2、ionsupportingsystemandonlycorrectdatacanproducecorrectresult，sohowtoinsuredataqualityofETLisveryimportant．Byanalyzingtheincrementaldataintegrationbasedontrigger，thisthesisf．oundthattraditionalmethod’sconcurrentoperationsonsnaptablemayleadrecordsinsnaptabletobeinValidandcannotcorrectlyrenecttheupd

3、atehistoryofdata，sodataatsourcewilldif．ferfromdataatdestinationandthereforeaf’fectthedataqua“tyofETL．Refbrringtotheconceptofsynchronizationinoperationsystem，thisthesissynchronizedconcurrentoperationsbyaddinglockfieldtosnaptable．Asaresult，therecordsofsnaptableareguaranteedtobecorrectanddataquality

4、ofETLunderincrementaldataintegrationbasedontriggerisalsoimproVed．Ontheotherhand，datawarehouseneedsETLtobeefficiencytoguaranteethetimelinessofdataandinordertosatisf．ythecomplicatedandchange2Lblebusinessrequirement，ETLhastobeagileonapplicabilityanddeVelopment．BydeeplyresearchingSEDA(StagedEvent-Dri

5、venArchitecture)andETLprocess，thisthesisconstructsETLprocessonSEDA．Firstly，comparingtomultithreadedETL，SEDAETLisprovedtobease伍ciencyasmultithreadedETLandproVidewaysf．orprecislytuning，dataprioritysupponingandloadcontrolling，grea气1yimproVesapplicabilityofETL．Second，SEDAETLusesstageasbasicexecutionu

6、nitinsteadofwholeETLprocess．Themodularityofstagemakesthecomplicatedresourcesmanagementtransparent，endusersonlyneedtoconcentratebusinesslogicwhendosecondarydevelopment．Atthesametime，thereusablityofstageproVidesthebasisforrapidlyconstructingETLapplicationandshoneningthedeVelopmenttimesofETL，reducin

7、gthedevelopmentcost．KeyWords：ExtractionTransfIormationLoading；StagedEVent-D“VenArchitecture；IncrementalDataExtractionIIl⋯⋯⋯⋯⋯．I⋯⋯⋯⋯．．II⋯⋯⋯⋯．III⋯⋯⋯⋯．VI⋯⋯⋯．．VII⋯⋯⋯⋯⋯1．．．．．．．．．．．．．．．1⋯⋯⋯⋯⋯2⋯⋯⋯⋯⋯41．5论文组织结构⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯