Conditional Random Fields を用いた日本語形態素解析.ppt

《Conditional Random Fields を用いた日本語形態素解析.ppt》由会员上传分享，免费在线阅读，更多相关内容在应用文档-天天文库。

1、ApplyingConditionalRandomFieldstoJapaneseMorphologicalAnalysisTakuKudo1*,KaoruYamamoto2,YujiMatsumoto11NaraInstituteofScienceandTechnology2CREST,TokyoInstituteofTechnology*Currently,NTTCommunicationScienceLabs.1BackgroundsConditionalRandomFields[Lafferty01]AvariantofMarkovRandomFi

2、eldsManyapplicationsPOStagging[Lafferty01],shallowparsing[Sha03],NErecognition[McCallum03],IE[Pinto03,Peng04]JapaneseMorphologicalAnalysisMustcopewithwordsegmentationMustincorporatemanyfeaturesMustminimizetheinfluenceofthelengthbias2JapaneseMorphologicalAnalysiswordsegmentation(noe

3、xplicitspacesinJapanese)POStagginglemmatization,stemmingINPUT:東京都に住む(IliveinMetropolisofTokyo.)東京/都/に/住む東京(Tokyo)NOUN-PROPER-LOC-GENERAL都(Metro.)NOUN-SUFFIX-LOCに(in)PARTICLE-GENERAL住む(live)VERBBASE-FORM3SimpleapproachforJMACharacter-basedbegin/insidetaggingnonstandardmethodinJMAcan

4、notdirectlyreflectlexiconsover90%accuracycanbeachievedusingthenaïvelongestprefixmatchingwithalexicondecodingisslow東京/都/に/住むBIBBBI4OurapproachforJMAAssumethatalexiconisavailablewordlatticerepresentsallcandidateoutputsreducesredundantoutputsUnknownwordprocessinginvokedwhennomatchingw

5、ordcanbefoundinalexiconcharactertypese.g.,Chinesecharacter,hiragana,katakana,number..etc5ProblemSettingInput:“東京都に住む(IliveinMetropolisofTokyo)”BOS東(east)[noun]東京(Tokyo)[noun]京都(Kyoto)[noun]都(Metro.)[suffix]に(in)[particle]に(resemble)[verb]住む(live)[verb]EOSLattice:京(capital)[noun]にpa

6、rticle,verb東noun京　noun東京　noun京都noun…GOAL:selecttheoptimalpathoutofallcandidatesInput:Output:lexiconNOTE:thenumberoftokens#Yvaries6Long-standingProblemsinJMA7ComplextagsetHierarchicaltagsetHMMscannotcapturethemHowtoselectthehiddenclasses?TOPlevel→lackofgranularityBottomlevel→dataspa

7、rsenessSomefunctionalparticlesshouldbelexicalizedSemi-automatichiddenclassselections[Asahara00]京都(Kyoto)NounProperLocGeneralKyoto8Complextagset,cont.Mustcaptureavarietyoffeatures京都(Kyoto)nounproperlocgeneralKyotoに(in)particlegeneralφφに住む(live)verbindependentφφlivebase-formPOShierar

8、chyoverlappingfeaturesinfl