资源描述：
《Crowdsourcing Performance Evaluations of User Interfaces用户界面的众包绩效评估》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、CrowdsourcingPerformanceEvaluationsofUserInterfacesStevenKomarov,KatharinaReinecke,KrzysztofZ.GajosIntelligentInteractiveSystemsGroupHarvardSchoolofEngineeringandAppliedSciences33OxfordSt.,Cambridge,MA02138,USAfkomarov,reinecke,kgajosg@seas.harvard.eduABSTRACTsu
2、bjectsresearch.ResearchersaredrawntoMTurkbecauseOnlinelabormarkets,suchasAmazon’sMechanicalTurktherelativeeaseofrecruitmentaffordslarger-scaleexperi-(MTurk),provideanattractiveplatformforconductinghu-mentation(intermsofthenumberofconditionstestedandmansubjectsex
3、perimentsbecausetherelativeeaseofre-thenumberofparticipantspercondition),afasterexperimen-cruitment,lowcost,andadiversepoolofpotentialpartici-talrevisioncycle,andpotentiallygreaterdiversityofpartici-pantsenablelarger-scaleexperimentationandfasterexperi-pantscomp
4、aredtowhatistypicalforlab-basedexperimentsmentalrevisioncyclecomparedtolab-basedsettings.How-inanacademicsetting[15,24].ever,becausetheexperimentergivesupthedirectcontrolThedownsideofsuchremoteexperimentationisthatthere-overtheparticipants’environmentsandbehavio
5、r,concernssearchersgiveupthedirectsupervisionoftheparticipants’aboutthequalityofthedatacollectedinonlinesettingsarebehaviorandthecontrolovertheparticipants’environments.pervasive.Inthispaper,weinvestigatethefeasibilityofInlab-basedsettings,thedirectcontactwithth
6、eexperimenterconductingonlineperformanceevaluationsofuserinterfacesmotivatesparticipantstoperformasinstructedandallowsthewithanonymous,unsupervised,paidparticipantsrecruitedexperimentertodetectandcorrectanybehaviorsthatmightviaMTurk.Weimplementedthreeperformance
7、experimentscompromisethevalidityofthedata.Becauseremotepartici-tore-evaluatethreepreviouslywell-studieduserinterfacede-pantsmaylackthemotivationtofocusonthetask,ormaybesigns.Weconductedeachexperimentbothinlabandonlinemoreexposedtodistractionthanlab-basedparticip
8、ants,con-withparticipantsrecruitedviaMTurk.Theanalysisofourcernsaboutthequalityofthedatacollectedinsuchsettingsresultsdidnotyieldanyevidenceofsignificantorsubstan-arep
