performance tuning profiling and concurrency

《performance tuning profiling and concurrency》由会员上传分享，免费在线阅读，更多相关内容在工程资料-天天文库。

1、PerformanceTuning,Profiling,andConcurrency"Prematureoptimizationistherootofallevil"–DonaldKnuth,arenownedcomputerscientistandmathematicianIntherealworld,therearemoreimportantthingsthanperformance,suchasfeatures,robustness,maintainability,testability,andusability.That'soneofthereasonsthatwedelayed

2、discussingthetopicofperformanceuntilthelastchapterofthebook.Wewillgivehintsonimprovingperformancewithprofilingasthekeytechnique.Formulticore,distributedsystems,wewilldiscusstherelevantframeworkstoo.Wewilldiscussthefollowingtopicsinthischapter:•Profilingthecode•InstallingCython•CallingtheCcode•Cre

3、atingapoolprocesswithmultiprocessing•SpeedingupembarrassinglyparallelforloopswithJoblib•ComparingBottlenecktoNumPyfunctions•PerformingMapReducewithJug•InstallingMPIforPython•IPythonParallelPerformanceTuning,Profiling,andConcurrencyProfilingthecodeProfilingisaboutidentifyingpartsofthecodethataresl

4、oworusealotofmemory.Wewillprofileamodifiedversionofthesentiment.pycodefromChapter9,AnalyzingTextualDataandSocialMedia.Thecodeisrefactoredtocomplywithmultiprocessingprogrammingguidelines.Youwilllearnaboutmultiprocessinglaterinthischapter.Also,wesimplifiedthestopwordsfiltering.Thethirdchangeistohav

5、efewerwordfeaturesasthereductiondoesn'timpactaccuracy.Thislastchangehasthemostimpact.Theoriginalcoderanforabout20seconds.Thenewcoderunsfasterthanthatandwillserveasthebaselineinthischapter.Somechangeshavetodowithprofilingandwillbeexplainedlaterinthissection.Pleaserefertotheprof_demo.pyfileinthisbo

6、ok'scodebundle:importrandomfromnltk.corpusimportmovie_reviewsfromnltk.corpusimportstopwordsfromnltkimportFreqDistfromnltkimportNaiveBayesClassifierfromnltk.classifyimportaccuracyfromlprof_hackimportprofile@profiledeflabel_docs():docs=[(list(movie_reviews.words(fid)),cat)forcatinmovie_reviews.cate

7、gories()forfidinmovie_reviews.fileids(cat)]random.seed(42)random.shuffle(docs)returndocs@profiledefisStopWord(word):returnwordinsworlen(word)==1@profiledeffilter_corpus():review_words=movie_reviews.words()print"#Review