资源描述：
《Design and Implementation of Relevance Assessments Using Crowdsourcing基于众包的关联性评估的设计与实现》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、DesignandImplementationofRelevanceAssessmentsUsingCrowdsourcingOmarAlonso1andRicardoBaeza-Yates21MicrosoftCorp.,MountainView,California,USAomar.alonso@microsoft.com2Yahoo!Research,Barcelona,Spainrbaeza@acm.orgAbstract.Inthelastyearscrowdsourcinghasemergedas
2、aviableplat-formforconductingrelevanceassessments.Themainreasonbehindthistrendisthatmakespossibletoconductexperimentsextremelyfast,withgoodresultsandatlowcost.However,likeinanyexperiment,thereareseveraldetailsthatwouldmakeanexperimentworkorfail.Togatherusef
3、ulresults,userinterfaceguidelines,inter-agreementmetrics,andjus-tificationanalysisareimportantaspectsofasuccessfulcrowdsourcingex-periment.InthisworkweexplorethedesignandexecutionofrelevancejudgmentsusingAmazonMechanicalTurkascrowdsourcingplatform,introducin
4、gamethodologyforcrowdsourcingrelevanceassessmentsandtheresultsofaseriesofexperimentsusingTREC8withafixedbudget.OurfindingsindicatethatworkersareasgoodasTRECexperts,evenprovidingdetailedfeedbackforcertainquery-documentpairs.Wealsoexploretheimportanceofdocument
5、designandpresentationwhenper-formingrelevanceassessmenttasks.Finally,weshowourmethodologyatworkwithseveralexamplesthatareinterestingintheirown.1IntroductionIntheworldoftheWeb2.0andusergeneratedcontent,oneimportantsub-classofpeercollaborativeproductionisthep
6、henomenonknownascrowdsourcing.Incrowdsourcing,potentiallylargejobsarebrokenintomanysmalltasksthatarethenoutsourceddirectlytoindividualworkersviapublicsolicitation.OneofthebestexamplesisWikipedia,whereeachentryorpartofanentrycouldbeconsideredasataskbeingsoli
7、cited.Asinthelaterexample,workerssometimesdoitforfree,motivatedeitherbecausetheworkisfunorduetosomeformofsocialreward[1,12].However,successfulexamplesofvolunteercrowdsourcingaredifficulttoreplicate.Asaresult,crowdsourcingincreasinglyusesafinancialcompensation,
8、usuallyasmicro-paymentsoftheorderofafewcentspertask.ThisisthemodelofAmazonMechanicalTurk(AMT1),wheremanytaskscanbedonequicklyandcheaply.AMTiscurrentlyusedasafeasiblealternativeforconductingallkindofrel
