Waxa aad akhrineyso xiga

• Horudhac (qaybta 4.1)

Su'aalaha ku saabsan sababaha cilmi-baarista bulshada badanaa waa kuwo adag oo adag. Si loo helo habdhaqan ku salaysan niyadda sababtoo ah garaafka natiijooyinka, eeg Pearl (2009) , iyo hab ku salaysan hab ku salaysan natiijooyinka iman kara, eeg Imbens and Rubin (2015) . Marka la barbardhigo labadan waji, fiiri Morgan and Winship (2014) . Si toos ah oo lagu qeexayo dhibaatada, fiiri VanderWeele and Shpitser (2013) .

Cutubkan dhexdiisa, waxaan abuuray waxa u muuqda sida laydh ilays ah oo u dhexeeya kartideena inaan ku qiyaaso qiyaasta natiijooyinka tijaabada ah iyo kuwa aan tijaabada ahayn. Si kastaba ha ahaatee, waxaan u maleynayaa, in xaqiiqda, farqiga udhexeeya ayaa ka sii daraya. Tusaale ahaan, qof kastaa wuxuu aqbalaa sigaarcabku inuu keeno kansar, xitaa haddii aan la tijaabin tijaabin la hubiyay oo dadka ku qasbaya in ay sigaar cabbaan had iyo jeer. Shadish, Cook, and Campbell (2001) ee dhererka ah ee lagu sameeyo qiyaasaha natiijooyinka aan natiijooyinka tijaabada ah ka eeg Rosenbaum (2002) , ( ??? ) , Shadish, Cook, and Campbell (2001) , iyo Dunning (2012) .

Cutubyada 1aad iyo 2aad ee Freedman, Pisani, and Purves (2007) si cad u qeexaya farqiga u dhexeeya tijaabooyinka, tijaabooyinka tijaabada ah, iyo tijaabooyinka la xakameeyey.

Manzi (2012) wuxuu bixiyaa hordhac cajiib ah oo la aqrisan karo falsafada iyo tirakoobyada ee tijaabooyinka la xakameeyey. Waxay sidoo kale bixisaa tusaalooyin xiiso leh oo dhab ah oo caalami ah ee awoodda tijaabinta ee ganacsiga. Issenberg (2012) wuxuu bixiyaa hordhac xiiso leh isticmaalka tijaabinta ololaha siyaasadeed.

• Waa maxay tijaabooyinka? (qaybta 4.2)

Box, Hunter, and Hunter (2005) , @ casella_statistical_2008, iyo Athey and Imbens (2016b) waxay bixiyaan Athey and Imbens (2016b) wanaagsan oo ku saabsan qaababka tirakoobka naqshadeynta iyo falanqaynta. Intaa ka sokow, waxaa jira daaweyn aad u fiican oo la adeegsanayo tijaabooyin dhinacyo kala duwan: dhaqaalaha (Bardsley et al. 2009) , cilmiga bulshada (Willer and Walker 2007; Jackson and Cox 2013) , cilmi nafsiga (Aronson et al. 1989) , sayniska siyaasadeed (Morton and Williams 2010) , iyo siyaasadda bulshada (Glennerster and Takavarasha 2013) .

Muhiimadda qorista kaqeybgalayaasha (tusaale ahaan, sameysashada) inta badan waxaa lagu qiimeeyaa cilmibaaris tijaabo ah. Si kastaba ha noqotee, haddii saameynta daaweyntu ay tahay mid aad uwanaagsan dadka, markaas tijaabada waa muhiim. Longford (1999) wuxuu si cad u tilmaamayaa marka uu u doodayo cilmi-baarayaashu ka fekerayaan tijaabooyinka sida sahan dadweyne oo leh sawir-qaadid.

• Laba sawir oo tijaabooyin ah: shaybaar iyo shaybaar siman (qaybta 4.3)

Waxaan soo jeediyay in ay jirto joogitaan u dhexeeya tijaabooyinka shaybaarka iyo goobaha cilmi-baarista, cilmi-baarayaasha kalena waxay soo jeediyeen noocyo faahfaahsan oo faahfaahsan, gaar ahaan kuwa kala soocaya noocyada kala duwan ee tijaabooyinka gudaha (Harrison and List 2004; Charness, Gneezy, and Kuhn 2013) .

Qaar badan oo waraaqo ah ayaa la barbardhigay tijaabooyinka shaybaarka iyo tijaabooyinka gudaha (Falk and Heckman 2009; Cialdini 2009) iyo marka la eego natiijooyinka tijaabo gaar ah ee sayniska siyaasadeed (Coppock and Green 2015) , dhaqaalaha (Levitt and List 2007a, 2007b; Camerer 2011; Al-Ubaydli and List 2013) , iyo cilmi nafsiga (Mitchell 2012) . Jerit, Barabas, and Clifford (2013) bixiyaan naqshad cilmi baaris oo wanaagsan oo isbarbardhigaya natiijooyinka shaybaarka iyo tijaabada goobta. Parigi, Santana, and Cook (2017) ayaa sharaxaya sida tijaabooyinka sahaminta internetka ay isugu dari karaan qaar ka mid ah astaamaha shaybaarka iyo tijaabooyinka goobta.

Cabashooyinka ku saabsan ka qaybgalayaashu bedelayaan dabeecadooda, sababtoo ah waxay ogyihiin in si dhow loo arkay waxaa mararka qaarkood loogu yeedhaa saameyn isweydaarsi , waxaana lagu baranayey cilmi nafsiga (Orne 1962) iyo dhaqaalaha (Zizzo 2010) . Inkasta oo inta badan lala xiriiriyo tijaabooyinka shaybaarka, arrimahan oo kale waxay keeni karaan dhibaatooyin tijaabo ah. Dhab ahaantii, saamaynta dalbashada waxaa sidoo kale loo yaqaan ' Hawthorne effects' , oo ah erey ka dhigaysa tijaabooyin caan ah oo caan ah oo ka bilaabmay 1924-dii hawlihii Hawthorne Works ee shirkadda Western Electric (Adair 1984; Levitt and List 2011) . Labada saameyn ee dalbashada iyo saameynta hawthorne waxay si aad ah ula xiriiraan fikradda cabbiraadda celceliska ah ee looga hadlay cutubka 2 (eeg sidoo kale Webb et al. (1966) ).

Imtixaanaadka gudaha waxay leeyihiin taariikh dheer oo ku saabsan dhaqaalaha (Levitt and List 2009) , sayniska siyaasadeed (Green and Gerber 2003; Druckman et al. 2006; Druckman and Lupia 2012) , cilmi nafsiga (Shadish 2002) , iyo siyaasadda dadweynaha (Shadish and Cook 2009) . Mid ka mid ah sayniska bulshada oo tijaabooyin sahlan ay si dhakhso ah u noqdeen horumarin caalami ah. Si dib-u-eegis waxtar leh oo ku saabsan shaqadaas oo ka mid ah dhaqaalaha wuxuu eegayaa Banerjee and Duflo (2009) , iyo qiimeyn culus eeg Deaton (2010) . Si dib-u-eegis loogu sameeyo shaqadan siyaasadda sayniska, arag Humphreys and Weinstein (2009) . Ugu dambeyntii, caqabadaha anshaxa ee ka yimaadda tijaabooyinka gudaha ayaa lagu soo bandhigay qaabka sayniska siyaasadeed (Humphreys 2015; Desposato 2016b) iyo dhaqaalaha horumarka (Baele 2013) .

Qeybtaan, waxaan soo jeediyay in macluumaadka daaweynta hore loo isticmaali karo si loo hagaajiyo qiyaasta daaweynta qiyaasta daaweynta, laakiin waxaa jira doodo ku saabsan habkan; Fiiri Freedman (2008) , W. Lin (2013) , Berk et al. (2013) , iyo Bloniarz et al. (2016) macluumaad dheeraad ah.

Ugu dambeyntii, waxaa jira laba nooc oo tijaabo ah oo ay sameeyaan saynisyahanno bulsheed oo aan ku habboonayn cabbirka shaybaarka: tijaabooyinka sahanka iyo tijaabooyinka bulshada. Daraasada sahanka ayaa ah tijaabooyin isticmaalaya kaabayaasha sahanada hadda jira oo isbarbardhigaya jawaabaha noocyo kale oo isku mid ah su'aal isku mid ah (tijaabooyinka sahan ee sahanka ayaa lagu soo bandhigay Cutubka 3aad); Wixii dheeraad ah oo ku saabsan tijaabada sahanka waxa ay u arkaan Mutz (2011) . Dhibaatooyinka bulsheed waa tijaabo ah halka daaweynta ay tahay siyaasad bulsho oo kaliya oo ay fulin karto dawlad kaliya. Dhibaatooyinka bulsheed waxay la xiriiraan qiimaynta barnaamijka. Wixii dheeraad ah oo ku saabsan tijaabada siyaasadda, fiiri Heckman and Smith (1995) , Orr (1998) , iyo @ glennerster_running_2013.

• Ka guuritaanka wixii ka baxsan tijaabooyinka fudud (qaybta 4.4)

Waxaan doortay in aan diiradda saaro saddex fikradood: ansaxnimada, ficillada waxyeellooyinka daaweynta, iyo hababka. Fikradahaas waxay leeyihiin magacyo kala duwan oo ku kala duwan qaybaha kala duwan. Tusaale ahaan, khabiirada cilmu nafsiga waxay u muuqdaan kuwo ka baxsan tijaabooyinka sahlan oo iyaga oo diirada saaraya dhexdhexaadiyeyaasha iyo wakiillada (Baron and Kenny 1986) . Fikradaha dhexdhexaadiyeyaasha waxaa lagu qabtaa waxa aan ugu yeerayo hababka, fikradda maareyayaasha waxaa lagu qabtaa waxa aan ugu yeerayo ansaxinta dibadda (tusaale ahaan, natiijooyinka tijaabadu way ka duwanaan laheyd haddii ay xaalado kala duwan ku dhexjiraan) iyo heerarka daaweynta ee saamaynta daaweynta ( Tusaale ahaan, waa saamaynta weyn ee dadka qaar ka badan kuwa kale).

Tijaabada ay sameeyeen Schultz et al. (2007) waxay muujinaysaa sida fikradaha bulshada loo isticmaali karo in loo habeeyo waxqabadyo waxtar leh. Wixii dood guud ah oo ku saabsan kaalinta aragtida ah ee dejinta waxqabadyo waxtar leh, fiiri Walton (2014) .

• Xaqdarnimada (qaybta 4.4.1)

Mabaadii'da asaasiga ah ee gudaha iyo dibaddaba waxaa markii hore soo bandhigay Campbell (1957) . Fiiri Shadish, Cook, and Campbell (2001) si aad u faahfaahsan taariikhda iyo faahfaahin taxaddar leh oo ku saabsan ansaxinta natiijooyinka tirakoobka, ansaxnimada gudaha, dhisida saxda ah, iyo ansaxinta dibadda.

Wixii guud ee arimaha la xidhiidha ansixinta natiijooyinka tirakoobka ee tijaabooyinka waxay Gerber and Green (2012) (laga eego dhinaca sayniska bulshada) iyo Imbens and Rubin (2015) ). Arrimaha qaarkood ee ku-ansaxinta natiijada tirakoobka ee ka soo baxa tijaabooyinka (Bakshy and Eckles 2013) ee internetka waxaa ka mid ah arrimaha sida hababka xisaabeed ee wax ku oolka ah ee loogu talagalay abuurista kalsoonida macluumaadka ku xiran (Bakshy and Eckles 2013) .

Ansaxinta gudaha waxay adkaan kartaa in la hubiyo in tijaabooyin adag oo adag. Eeg Tusaale ahaan, Gerber and Green (2000) , Imai (2005) , iyo Gerber and Green (2005) si ay uga doodaan hirgelinta tijaabo sahlan oo ku saabsan cod bixinta. Kohavi et al. (2012) iyo Kohavi et al. (2013) waxay soo bandhigaysaa hirgelinta caqabadaha ku-oolka ah ee ku-meel-gaadhka ah ee tijaabooyinka gudaha.

Mid ka mid ah halista ugu weyn ee ku-ansaxinta gudaha waa suurtagalnimada in la kala sooco. Hal dariiq oo suurtagal ah oo lagu ogaanayo dhibaatooyinka kala soocidda waa in la isbarbardhigo kooxaha daaweynta iyo kooxaha xakamaynta siyaabaha la arki karo. Isbarbardhigga noocan ah waxaa lagu magacaabaa jeeg dheellitirnaanta . Fiiri Hansen and Bowers (2008) si loo eego habka xisaabinta miisaaniyada iyo Mutz and Pemantle (2015) si ay uga walwalaan hubinta miisaaniyadda. Tusaale ahaan, iyadoo la adeegsanayo jeegareynta miisaaniyadda, Allcott (2011) heshay caddaymo muujinaya in kala soocida aan si sax ah loo fulin seddex ka mid ah tijaabooyinka Opower (fiiri jadwalka 2, goobaha 2, 6, iyo 8). Xeeladaha kale, fiiri cutubka 21 ee Imbens and Rubin (2015) .

Waxyaabaha kale ee muhiimka ah ee la xidhiidha ansaxinta gudaha waxaa ka mid ah: (1) mid aan dhinac aheyn, oo aan qof kasta oo kooxda daaweyntu dhab ahaantii helin daaweynta, (2) laba dhinac oo aan dhinac aheyn, oo aan qof kastoo ka tirsan kooxda daaweyntu helayaan daaweynta iyo dadka qaarkood Kooxda kantaroolka ayaa helaysa daaweynta, (3) fara-galin, halka natiijooyinka aan la qiyaasin ka qaybgalayaasha qaarkood, iyo (4) faragelinta, halkaas oo daaweyntu ay ka soo baxaan dadka ku jira xaalad daaweynta dadka ku sugan xaaladda xakamaynta. Eeg cutubyada 5, 6, 7, iyo 8 ee Gerber and Green (2012) si aad uga heshid arrimahan mid kasta.

Faahfaahin dheeraad ah oo ku saabsan sidii loo dhisi lahaa waxqabad, eeg Westen and Rosenthal (2003) , iyo waxyaabo badan oo ku saabsan in la dhiso wax ku ool ah ilaha xogta waaweyn, Lazer (2015) iyo cutubka 2 ee buuggan.

Hal dhinac oo ka mid ah ansaxinta dibadda ayaa ah meesha ay tahay in la baaro. Allcott (2015) wuxuu bixiyaa daaweyn qoto-dheer oo xeeldheer ah oo ku saabsan qaabka xulashada goobta. Arrintan waxaa sidoo kale laga wada hadlay Deaton (2010) . Muuqaal kale oo lagu ansaxiyo dibadda ayaa ah haddii hawlgallada kale ee wax ka-qabashada ahi ay yeelan doonaan saameyn isku mid ah. Xaaladdan, marka la barbar dhigo Schultz et al. (2007) iyo Allcott (2011) waxay muujinaysaa in tijaabooyinka Opower ay leeyihiin saameyn la qiimeeyey oo la qiimeeyay intii ay tijaabooyinkii hore ee Schultz iyo asxaabta (1.7% 5%). Allcott (2011) qanacsan tahay in tijaabooyinka soo socda ay saameyn yar ku yeesheen sababtoo ah siyaabaha daaweyntu u kala duwan yihiin: qoraallada elektarooniga ah oo qayb ka ah daraasad ay maalgelisay jaamacad, marka la barbar dhigo qoraallo daabacan oo qayb ka ah mass-produced warbixinta shirkadda tamarta.

• Hoos u dhigga saameynta daaweynta (qaybta 4.4.2)

Si aad u aragto aragtid aad u wanaagsan oo ku saabsan nafsadda ee saameynta daaweynta ee tijaabooyinka goobaha, fiiri cutubka 12 ee Gerber and Green (2012) . Wixii ku saabsan hordhacnimada hannaanka daaweynta ee tijaabooyinka caafimaadka, fiiri Kent and Hayward (2007) , Longford (1999) , iyo Kravitz, Duan, and Braslow (2004) . Tixgelinnada hiddaha ee waxyeellooyinka daaweynta guud ahaan waxay diiradda saaraan kala duwanaanta ku saleysan sifooyinka daaweynta ka hor. Haddii aad xiiseyneyso heterogaannimada ku saleysan natiijooyinka daaweynta ka dib, markaa waxaa loo baahan yahay qaabab badan oo adag, sida habka aasaasiga ah (Frangakis and Rubin 2002) ; eeg Page et al. (2015) dib u eegis.

Cilmi-baareyaasha badani waxay qiyaasaan hiddaha hannaanka daaweynta iyadoo la adeegsanayo nidaam joogto ah, laakiin hababka cusub ayaa ku tiirsan barashada mashiinka; eeg, tusaale, Green and Kern (2012) , Imai and Ratkovic (2013) , Taddy et al. (2016) , iyo Athey and Imbens (2016a) .

Waxaa jira shaki ah oo ku saabsan natiijooyinka helitaanka saamaynta hiddaha waxyeellada sababta oo ah dhibaatooyinka isbarbardhigga badan iyo "kalluumeysiga." Waxaa jira habab kala duwan oo tirakoob oo kaa caawin kara wax ka qabashada cabashooyinka ku saabsan isbarbardhigga isbarbardhiga (Fink, McConnell, and Vollmer 2014; List, Shaikh, and Xu 2016) . Qorshaha hore ee dareenka ku saabsan "kalluumeysiga" waa diiwaangelin hor leh, taas oo noqotey mid sii kordhaysa ee cilmi nafsiga (Nosek and Lakens 2014) , sayniska siyaasadeed (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , iyo dhaqaalaha (Olken 2015) .

Daraasada ay sameeyeen Costa and Kahn (2013) kaliya qiyaastii kala bar qoysaska tijaabada ah waxay ku xirnaan karaan macluumaadka dadweynaha. Akhriyeyaashu daneynaya faahfaahintan waa inay tixraacaan warqadda asalka ah.

• Mechanisms (qaybta 4.4.3)

Mashruucyada waa wax aad u muhiim ah, laakiin waxay u soo jeestaan ​​inay adagtahay inay wax bartaan. Cilmi-baaris ku saabsan qaababku waxay la xiriiraan daraasaddaha dhexdhexaadiyeyaasha ee cilmi-nafsiga (laakiin sidoo kale fiiri VanderWeele (2009) si isbarbar dhidid ah u dhexeeya labada fikradood). Hababka tirakoobka ee helitaanka hababka, sida habka loo horumariyo Baron and Kenny (1986) , waa wax caadi ah. Nasiib darro, waxay soo baxday in nidaamyadani ay ku xiran tahay fikrado xooggan (Bullock, Green, and Ha 2010) waxayna la dhibaan marka ay jiraan qaabab badan, sida mid ka mid ah laga filayo xaalado badan (Imai and Yamamoto 2013; VanderWeele and Vansteelandt 2014) . Imai et al. (2011) iyo Imai and Yamamoto (2013) bixiyaan qaar ka mid ah hababka tirakoobka. Dheeraad ah, VanderWeele (2015) waxay bixisaa daaweyn mudo-dheer oo leh dhowr natiijooyin oo muhiim ah, oo ay ku jiraan habab dhammaystiran oo lagu falanqeyn karo dareenka.

Nidaam gaar ah wuxuu diiradda saaraa tijaabooyinka isku dayaya in lagu hagaajiyo qaabka si toos ah (tusaale ahaan, in la siiyo vitamin C badmaaxaha). Nasiib darro, goobaha sayniska ee badan, waxaa jira habab badan oo ay adag tahay in la naqshadeeyo daaweynta isbeddelaysa mid aan is beddelaynin kuwa kale. Qaar ka mid ah qaababka isbeddelka isbeddelka waxaa lagu sharaxay Imai, Tingley, and Yamamoto (2013) , Ludwig, Kling, and Mullainathan (2011) , iyo Pirlott and MacKinnon (2016) .

Cilmi-baareyaasha si buuxda u tijaabinaya tijaabooyinka baaritaanka waxay u baahan doonaan inay ka walwalaan imtixaanno badan oo lagu baaro; eeg Fink, McConnell, and Vollmer (2014) iyo List, Shaikh, and Xu (2016) si aad u hesho macluumaad dheeraad ah.

Ugu dambeyn, farsamooyinka waxay sidoo kale leeyihiin taariikh dheer oo falsafadda sayniska sida lagu qeexay Hedström and Ylikoski (2010) .

• Isticmaalka agagaaraha hadda jira (qaybta 4.5.1)

Wixii dheeraad ah ee ku saabsan isticmaalka daraasadaha waraaqaha iyo baaritaannada daraaseynta si loo qiyaaso takoorka, fiiri Pager (2007) .

• Dhismo tijaabadaada (qaybta 4.5.2)

Habka ugu badan ee loo qoro ka qaybgalayaasha tijaabooyinka aad dhisto waa Amazon Mechanical Turk (MTurk). Sababtoo ah MTurk moodada qaybaha tijaabooyinka shaybaadh-dhaqameedka-bixinaya dadka si ay u dhamaystiraan shaqooyinka ayan u qaban doonin cilmi-baarayaal bilaash ah ayaa durbadiiba bilaabay isticmaalka Turkers (shaqaalaha MTurk) inay yihiin kaqeybgalayaal tijaabo ah, taas oo keentay in ururinta xogta la dhaqsaha badan yahay iyo ka jaban halkii lagu gaari lahaa tijaabooyin tijaabo ah oo ku-meelgaar ah (Paolacci, Chandler, and Ipeirotis 2010; Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012; Rand 2012; Berinsky, Huber, and Lenz 2012) .

Guud ahaan, faa'iidada ugu weyn ee isticmaalida kaqeybgalayaasha ee MTurk waa sahayd. Halka tijaabooyinka shaybaadhku ay qaadan karaan todobaadyo si ay u ordo oo ay tijaabiyaan tijaabooyinka waxay qaadan karaan bilooyin si ay u sameeyaan, tijaabooyinka ka qaybgalayaasha laga soo qaatay MTurk ayaa lagu wadi karaa maalmo. Tusaale ahaan, Berinsky, Huber, and Lenz (2012) waxay awoodeen inay qoraan 400 maado hal maalin si ay uga qaybqaataan tijaabinta 8 daqiiqo. Dheeraad ah, kaqaybgalayaasha waxaa loo qori karaa wax kasta oo ujeedo ah (oo ay ku jiraan baaritaanada iyo iskaashiga ballaaran, sida ku xusan cutubyada 3 iyo 5). Fududeynta qorista ayaa ka dhigan in cilmi-baarayaashu ay isku dari karaan taxane ah tijaabooyin la xidhiidha isku-duwidda degdega ah.

Ka hor inta aanad qorin ka qaybgalayaasha ka socda MTurk ee tijaabadaada, waxa jira afar waxyaabo muhiim ah oo aad u baahan tahay inaad ogaato. Marka hore, cilmi-baarayaal badani waxay leeyihiin shaki aan shaki lahayn oo tijaabo ah oo ku lug leh Turkiil. Sababtoo ah shaki la'aantu maaha mid gaar ah, way adag tahay in lagala soo horjeedo caddaynta. Si kastaba ha noqotee, ka dib sannado badan oo daraasaddo ah oo isticmaalaya Turkers, waxaan hadda ka dhaadhicin karnaa in tani shaki ku jirin aan gaar ahaan la aqbali karin. Waxaa jira daraasado badan oo la barbardhigayey nooma kala soocida Turkiyiinta iyo dadka kale iyo daraasado badan oo la barbardhigay natiijooyinka tijaabooyinka Turkic ee ka soo jeeda dadka kale. Marka la eego dhammaan shaqadan, waxaan u maleynayaa in habka ugu wanaagsan ee aad u fekereyso in ay tahay in Turker ay tahay tusaale macquul ah, sida ardayda, laakiin wax yar oo kala duwan (Berinsky, Huber, and Lenz 2012) . Sidaa darteed, sida ardaydu u yihiin dad macquul ah qaar ka mid ah, laakiin dhammaantood, cilmi-baarista, Turkers waa dad macquul ah qaar ka mid ah, laakiin dhammaantood, cilmi-baaris. Haddii aad la shaqeyneyso Turkiyiinta, markaa waxaa macquul ah in aad akhrido qaar badan oo ka mid ah daraasaddan barbardhigga ah oo aad fahamsan tahay jumladahooda.

Marka labaad, cilmi-baarayaashu waxay sameeyeen hababkii ugu fiicnaa ee kor u qaadaya ansaxnimada gudaha ee tijaabooyinka MTurk, waana inaad barataa oo aad raacdaa hababka ugu wanaagsan (Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012) . Tusaale ahaan, cilmi-baarayaasha isticmaala Turkers waxaa lagu dhiirigelinayaa inay isticmaalaan qalabka wax lagu qoro si ay uga saaraan ka-qaybgalayaasha aan (Berinsky, Margolis, and Sances 2014, 2016) (laakiin sidoo kale arag DJ Hauser and Schwarz (2015b) iyo DJ Hauser and Schwarz (2015a) ). Haddii aadan ka saarin ka-qaybgalayaasha aan-laheeyn, wixii waxyeello ah ee daaweyntu waa la dhaqaajin karaa dhawaaqa ah ee ay soo bandhigi kareen, iyo sida dhabta ah ee ka qaybgalayaasha aan walbahaarin waxay noqon karaan kuwo la taaban karo. Imtixaanka Huber iyo jalayaashiisa (2012) , qiyaastii 30% kaqaybgalayaasha ayaa ku guuldareystay falanqeeyayaasha muhiimka ah. Dhibaatooyinka kale ee caadi ahaan soo baxa marka la isticmaalo Turkiyiinta waa ka-qaybgalayaasha aan caadiga ahayn (Chandler et al. 2015) iyo fara- (Zhou and Fishbach 2016) .

Saddexaad, marka loo eego noocyo kale oo tijaabo ah oo tijaabo ah, tijaabooyinka MTurk ma gaari karaan; Stewart et al. (2015) qiyaasaya in waqti kasta oo la bixiyay ay jiraan kaliya qiyaastii 7,000 oo qof oo ku nool MTurk.

Ugu dambeyntii, waa inaad ogaataa in MTurk yahay bulsho leh xeerarkeeda iyo xeerarkeeda (Mason and Suri 2012) . Si isku mid ah oo aad isku dayi lahayd inaad ka ogaato dhaqanka waddanka aad u socon lahayd inaad tijaabiso tijaabooyinkaaga, waa inaad isku daydaa inaad ogaato wax badan oo ku saabsan dhaqanka iyo caadooyinka Turkiga (Salehi et al. 2015) . Waa inaad ogaataa in Turkiyiinta ka hadli doono tijaabintaada haddii aad samaynayso wax aan habooneyn ama aan caddaalad ahayn (Gray et al. 2016) .

MTurk waa hab cajiib ah oo ku habboon in lagu qoro kaqeybgalayaasha tijaabooyinkaada, ha noqdaan labadiinaba, sida Huber, Hill, and Lenz (2012) , ama wax ka badan sida kuwa Mason and Watts (2009) , Goldstein, McAfee, and Suri (2013) , Goldstein et al. (2014) , Horton and Zeckhauser (2016) , iyo Mao et al. (2016) .

• Dhis sheygaaga (qaybta 4.5.3)

Haddii aad ka fekereyso inaad isku daydo inaad abuurto sheygaaga, waxaan kugula talinayaa in aad akhrisid talo bixinta ay soo bandhigtay kooxda MovieLens ee Harper and Konstan (2015) . Fikrado muhiim ah oo ka soo baxa waayo-aragnimadooda waxay tahay in mashruuc kasta oo guulo leh uu jiro wax badan oo badan oo badan. Tusaale ahaan, kooxda MovieLens waxay soo saareen badeecooyin kale, sida GopherAnswers, kuwaas oo dhammaystiray guuldaradii (Harper and Konstan 2015) . Tusaale kale oo cilmi-baaris ah oo ku fashilmay isaga oo isku dayey in uu dhiso wax soo saarku waa Edward Castronova isku dayga ah inuu dhiso kulan online ah oo lagu magacaabo Arden. Inkastoo $ 250,000 oo maaliyadeed ah, mashruucu wuxuu ahaa mid furan (Baker 2008) . Mashruucyada sida GopherAnswers iyo Arden nasiib daro way ka badan yihiin mashaariicda sida MovieLens.

• Wada shaqeeyaha leh awoodda (qaybta 4.5.4)

Waxaan maqlay fikradda Pasteur's Quadrant si joogta ah ugala hadlo shirkadaha tiknoolajiyada, waxayna ka caawisaa isku-duwidda dadaalka cilmi-baarista Google (Spector, Norvig, and Petrov 2012) .

Bond iyo daraasada saaxiibada (2012) sidoo kale isku dayayaan inay ogaadaan saameynta daaweyntan ay ku hayaan asxaabta kuwa qaata. Sababtoo ah naqshadeynta tijaabada ah, waraaqahaas waa ay adag tahay in si nadiif ah loo ogaado; akhristayaasha xiisaha leh waa inay arkaan Bond et al. (2012) si aad uga wada hadashaan. Jones iyo jalayaashiisa (2017) sidoo kale sameeyay tijaabo isku mid ah intii lagu jiray doorashadii 2012. Tijaabooyinkaasi waxay qayb ka yihiin caadooyinka dheeraadka ah ee tijaabooyinka sayniska siyaasadeed ee ku saabsan dadaallada lagu dhiirigelinayo cod bixinta (Green and Gerber 2015) . Dhibaatooyinkan cod-bixinta ee caadiga ah ayaa ah mid caadi ah, qayb ahaan, sababtoo ah waxay ku jiraan Ku-xigeenka Pasteur. Taasi waxaa weeye, waxaa jira dad badan oo dhiirigalinaya kordhinta cod bixinta iyo codbixinta waxay noqon karaan dabeecad xiiso leh si loo tijaabiyo aragtida guud ee ku saabsan isbedelka habdhaqanka iyo saameynta bulshada.

Wixii talo ah ee ku saabsan tijaabooyinka tijaabada ah ee ururada iskaashiga ah sida xisbiyada siyaasadeed, NGO-yada iyo ganacsiyada, eeg Loewen, Rubenson, and Wantchekon (2010) , JA List (2011) , iyo Gueron (2002) . Fikradaha ku saabsan sida wada shaqaynta ururada ay saameyn karto naqshadaynta cilmi-baarista, eeg King et al. (2007) iyo Green, Calfano, and Aronow (2014) . Iskaashiga wuxuu sidoo kale u horseedi karaa su'aalo anshaxeed, sida lagu sheegay Humphreys (2015) iyo Nickerson and Hyde (2016) .

• Talo-soo-dejinta (qaybta 4.6)

Haddii aad sameysid qorshe falanqayn ah inta aanad tijaabin tijaabadaada, waxaan soo jeedinayaa inaad bilowdo adoo akhrinaya tilmaamaha warbixinta. Xeerarka CONSORT (Warbixinta Istaandarada Warbixinta Iskuduwida) ayaa lagu sameeyey daawo (Schulz et al. 2010) iyo isbedelka cilmi-baarista bulshada (Mayo-Wilson et al. 2013) . Tilmaamo la xidhiidha ayaa waxaa diyaariyay tifaftirayaasha Jaamicadda Sayniska Siyaasadeed ee Siyaasadda (Gerber et al. 2014) (eeg sidoo kale Mutz and Pemantle (2015) iyo Gerber et al. (2015) ). Ugu dambeyntii, warbixinta warbixinta ayaa lagu sameeyey cilmi nafsiga (APA Working Group 2008) , sidoo kale fiiri Simmons, Nelson, and Simonsohn (2011) .

Haddii aad sameysato qorshe falanqeyn, waa inaad tixgelisaa kahor diiwaangelinta sababta oo ah horay isdiiwaangelintu waxay kordhin doontaa kalsoonida ay dadka kale u leeyihiin natiijooyinkaaga. Dheeraad ah, haddii aad la shaqaynayso lamaane, waxay xaddidaysaa awooddaada lamaanahaaga si aad u bedesho falanqaynta kadib markaad aragto natiijooyinka. Diiwaangelinta hore ayaa sii kordheysa ee cilmi nafsiga (Nosek and Lakens 2014) , sayniska siyaasadeed (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , iyo dhaqaalaha (Olken 2015) .

Talooyinka naqshadeynta ee loogu talagalay tijaabooyinka internetka ayaa sidoo kale lagu soo bandhigayaa Konstan and Chen (2007) iyo Chen and Konstan (2015) .

Waxa aan ugu yeeray istiraatiijiyadda istiraatiijiga ah waxaa mararka qaarkood loo yaqaan cilmi-baarista barnaamijyada ; fiiri Wilson, Aronson, and Carlsmith (2010) .

• Abuuraan xogta qiimaha eber (variable 4.6.1)

Salganik, Dodds, and Watts (2006) , Salganik and Watts (2008) , Salganik and Watts (2009b) , Salganik and Watts (2009a) , iyo Salganik (2007) . Wixii dheeraad ah oo ku saabsan ku guuleysta-suuqyada oo dhan, fiiri Frank and Cook (1996) . Wixii faahfaahin dheeraad ah oo ku saabsan Nasiib darro iyo xirfad guud ahaan guud, eeg Mauboussin (2012) , Watts (2012) , iyo Frank (2016) .

Waxaa jira hab kale oo looga takhaluso lacagaha ka qaybgalayaasha kuwaas oo cilmi-baarayaashu ay isticmaalayaan taxadar: xannaaneyn. Dhamaan tijaabooyinka tijaabada ah ee internetka kaqaybgalayaasha waxaa aasaas u ah tijaabooyin oo aan waligood magdhabin. Tusaalooyinka habkan waxaa ka mid ah Restivo iyo van de Rijt (2012) tijaabinta abaalmarinta ee Wikipedia iyo Bond iyo saaxiibka (2012) tijaabada lagu dhiirigelinayo dadka inay codeeyaan. Tijaabooyinkaan dhab ahaantii ma laha qiime gaaban oo eber ah-halkii, waxay leeyihiin qiime-gaabin eber ah ee cilmi-baarayaasha . Tijaabooyinka noocan ah, xitaa haddii kharashka kaqaybgalaha kasta uu aad u yaryahay, qiimaha wadarta waxay noqon kartaa mid aad u weyn. Cilmi-baadhayaasha ku shaqeeya tijaabooyinka internet-ka ee badanaa waxay caddaynayaan muhiimada ay leedahay saameynta daaweynta qiyaasta yar ee adoo adeegsanaya iyaga oo sheegaya in saameyntan yaryar ay muhiim u noqon karto marka ay dad badan ku dhaqmaan. Xaqiiqda saxda ah ee khatarta ah waxay khuseysaa kharashyada ay cilmi-baarayaashu ku soo rogayaan kaq Haddii tijaabooyinkaada ay keento hal malyan oo qof in ay hal mar ka dhumiyaan, tijaabadu maaha mid halis u ah qof gaar ah, laakiin wadar ahaan labadii sano ee la soo dhaafay.

Hab kale oo lagu abuurayo kharashka ku-beddelaadaha eber-ka ee ka-qaybgalayaashu waa inay isticmaalaan bakhtiyaanasiib, hab loo isticmaalo cilmi-baarista (Halpern et al. 2011) . Wixii dheeraad ah oo ku saabsan qaabeynta khibradaha isticmaalka raaxada, fiiri Toomim et al. (2011) . Wixii intaa ka badan oo ku saabsan isticmaalka bots si aad u abuurto tijaabo ah kharashka isbeddelka eber ( ??? ) eeg ( ??? ) .

• Beddel, Iskudar, iyo Yaree (Qaybta 4.6.2)

Saddexda R ee asal ahaantii soo jeediyay by Russell and Burch (1959) waa sidan soo socota:

"Bedelida ka dhigan bedelay ee ka dhawrsada nool xayawaanka sare ee wax insentient. Dhimista ka dhigan tahay hoos u tirada xoolaha loo isticmaalo si aad u hesho macluumaad ah ee qadar siin iyo sax. Hadba waxay ka dhigan tahay hoos u dhac kasta oo dhacdooyinka ama adkaanta nidaamka ka baxsan codsatay in kuwa xoolaha oo weli in la isticmaalo. "

Seddexda R ee aan soo jeediyay ma ahan kuwa ka hor imanaya mabaadi'da anshaxa ee lagu sharraxay cutubka 6aad. Taas waxay ka dhigan tahay, mid ka mid ah mabaa'diida-faa'iidada-gaar ahaan goobaha tijaabada bani-aadamka.

Marka la eego R-da koowaad ("beddel"), marka la barbardhigo tijaabinta shucuurta niyadeed (Kramer, Guillory, and Hancock 2014) iyo tijaabinta dabiiciga ah ee (Lorenzo Coviello et al. 2014) ahaaneed (Lorenzo Coviello et al. 2014) waxay bixisaa casharo guud oo ku saabsan ganacsiyada markaad ka gudubto tijaabooyinka tijaabo dabiiciga ah (iyo habab kale oo la mid ah isku-darka isku-dayga ah ee isku-dayga tijaabada ah ee xogta aan cilmi-baarista ahayn, fiiri cutubka 2). Marka lagu daro faa'iidooyinka anshaxa, badalashada tijaabada ah ee daraasadaha aan cilmi-baarista ahayn waxay sidoo kale u sahlaysaa cilmi-baarayaasha in ay baraan daaweynyada in ay yihiin kuwo aan suurtagal ahayn in la diro. Hase yeeshee, faa'iidooyinka anshaxa iyo saad-bixinta waxay ku yimaadaan kharash, si kastaba ha ahaatee. Iyadoo tijaabooyin dabiici ah ay cilmi-baarayaashu koontaroolayaan waxyaabo ay ka mid yihiin qorista ka qaybgalayaasha, kala xulashada, iyo nooca daaweynta. Tusaale ahaan, hal xaddidaad oo ah roobab sida daaweyntu waxay tahay in labaduba ay kor u qaadaan waxqabadkooda iyo hoos u dhigista xasaasiyad. Daraasada tijaabada ah, Kramer iyo asxaabtu waxay awood u yeesheen in ay si madaxbannaani ah u hagaajiyaan natiijada iyo xasilloonida. Habka gaarka ah ee loo adeegsado Lorenzo Coviello et al. (2014) waxaa sii dheer L. Coviello, Fowler, and Franceschetti (2014) . Hordhac ku saabsan doorsoomayaasha qalabka, taas oo ah habka uu adeegsado Lorenzo Coviello et al. (2014) , fiiri Angrist and Pischke (2009) (aan rasmi ahayn) ama Angrist, Imbens, and Rubin (1996) ). Qiimaynta naxdinta leh ee isbeddelka qalabka, fiiri Deaton (2010) , iyo hordhaca qalabka isbeddelka leh qalabka daciifka ah (roobku waa qalab daciif ah), eeg Murray (2006) . Dunning (2012) , Rosenbaum (2002) , ( ??? ) , iyo Shadish, Cook, and Campbell (2001) bixiyaan fikrado wanaagsan oo ku saabsan qiyaasaha saameynta naafanimo iyadoo aan la tijaabin tijaabooyin.

Marka la eego R (labaad "R"), waxaa jira warshado farsamo iyo farsameyn ah marka la tixgelinayo bedelida naqshadaynta Fikradaha Dareenka ee ka horjoogsanaya posts si loo xoojiyo boostooyinka. Tusaale ahaan, waxaa dhici karta in farsamaynta farsamo ee "Feed Feed Feed" ay ka dhigto mid sahlan in la sameeyo tijaabo ah xarumaha ay xiranyihiin halkii ka mid ah oo ay ku xoojinayaan (xusuusnow in tijaabin la xidhiidha xayiraadda posts sida lakabka sare ee nidaamka Feed Feed News iyada oo aan loo baahnayn wax ka bedel nidaamka hoosta). Caqli ahaan, si kastaba ha ahaatee, aragtida ay ka hadlaysay tijaabadu si cad uma soo jeedin hal naqshad kale. Nasiib daro, anigu ma ogi baaritaano muhiim ah oo ku saabsan mudnaanta qaddarinta ee xannibista iyo kobcinta macluumaadka ee Feed Feeding. Sidoo kale, ma aanan arkin baaritaan badan oo ku saabsan daaweynta daaweynta si aan waxyeello u gaarsiin; hal B. Jones and Feamster (2015) waa B. Jones and Feamster (2015) , oo tixgelinaya kiisaska cabirka internet-ka (mawduuc aan kaga wada hadli karo cutubka 6aad ee xiriirka daraasadda Encore (Burnett and Feamster 2015; Narayanan and Zevenbergen 2015) ).

Marka la eego saddexda R ("dhimista"), barashada wanaagsan ee falanqaynta awoodda dhaqameed waxaa bixiya Cohen (1988) ) iyo Cohen (1992) (maqaal), halka Gelman and Carlin (2014) siiyaan aragti yar. Daawooyinka horay u-daaweynta waxaa lagu dari karaa marxaladda naqshada iyo falanqaynta ee tijaabooyinka; Cutubka 4 ee Gerber and Green (2012) ayaa si wanaagsan u soo bandhigaya labada waji, Casella (2008) waxay bixisaa daaweyn qoto dheer. Farsamooyinka isticmaalaya macluumaadkan kahor-daaweyneed ee kala-soocida waxaa caadi ahaan loo yaqaannaa qaababka tijaabada ah ee la xakameynayo ama naqshadaynta tijaabada ah ee qaabaysan (terminology looma isticmaalo si joogto ah bulshooyinka); farsamooyinkani waxay si dhow ula xiriiraan farsamooyinka tijaabada ah ee lagu falanqeeyay cutubka 3aad. Ka eeg Higgins, Sävje, and Sekhon (2016) si aad u isticmaashid waxyaabahaan tijaabada ah ee tijaabo badan. Daawooyinka horay loo daaweyn karo ayaa sidoo kale lagu dari karaa marxaladda falanqaynta. McKenzie (2012) waxay faahfaahinaysaa qaabka farqiga u dhexeeya ee kala duwan ee falanqaynta tijaabooyinka gudaha si faahfaahsan. Fiiri Carneiro, Lee, and Wilhelm (2016) wax badan oo ku saabsan suuqyada kala duwan ee u dhexeeya habab kala duwan si loo kordhiyo qiyaasta qiyaasaha daaweynta. Ugu dambeyntii, markaad go'aansato in aad isku daydo inaad ku darto tusaalooyinka daaweynta ka hor marxaladda naqshada ama falanqaynta (ama labadaba), waxaa jira dhowr arrimood oo la tixgelinayo. Meelaha ay cilmi-baarayaashu doonayaan inay muujiyaan inaanay ahayn "kalluumeysi" (Humphreys, Sierra, and Windt 2013) , iyada oo la adeegsanayo marxaladaha horudhaca daaweynta ee marxaladda naqshadeynta waxay noqon kartaa waxtar (Higgins, Sävje, and Sekhon 2016) . Xaaladaha ay ka qaybgalayaashu ku soo noqdaan, gaar ahaan tijaabooyinka goobaha internetka, iyagoo isticmaalaya macluumaadka daaweynta hore ee marxaladda naqshadeynta waxay noqon kartaa mid adag; tusaale ahaan, Xie and Aurisset (2016) .

Waxaa habboon in lagu daro waxoogaa ah dareen ah oo ku saabsan sababta habka farqiga u kala duwanaanta ay noqon karto wax aad u tayo badan marka loo eego isbeddel mid ah. Natiijooyin badan oo internetka ah ayaa leh kala duwanaansho aad u sarreeya (eeg tusaale ahaan, RA Lewis and Rao (2015) iyo Lamb et al. (2015) ) waxayna aad u deggan yihiin waqtigooda. Xaaladdan oo kale, dhibcaha isbeddelku waxay yeelan doonaan kala duwanaansho yar oo yar, kor u qaadista awooda baaritaanka tirakoobka. Mid ka mid ah sababahaas aan loo isticmaalin marar badan waa in ka hor da'da dijitalka, ma ahan wax caadi ah in la helo natiijooyinka daaweynta ka hor. Hab siyaab badan oo looga fekerayo tan waxa weeye in la qiyaaso tijaabinta si loo qiyaaso in jimicsi gaar ah uu sababo miisaanka. Haddii aad qaadatid nidaam farqiga u dhexeeya, qiyaastaadu waxay lahaan doontaa isbeddel ka yimaada miisaaniyadda miisaanka dadweynaha. Hase yeeshee, haddii aad samayso hab isbeddel ah oo isbeddel leh, si kastaba ha noqotee, isbeddel dabiiciga ah ee miisaanka ayaa laga saaraa, aadna si sahlan u ogaan kartaa faraqa u keenay daaweynta.

Ugu dambeyntii, waxaan ku fikiray inaan ku daro afaraad R: "dib u celi". Taasi waa, haddii cilmi-baarayaashu ay helaan macluumaad badan oo tijaabo ah marka loo eego waxay u baahan yihiin inay wax ka qabtaan su'aalahooda cilmi-baarista asalka ah, waa inay soo celiyaan xogta si ay u waydiiyaan su'aalo cusub. Tusaale ahaan, ka feker in Kramer iyo asxaabtu ay adeegsadeen qiyaasaha farqiga u dhexeeya iyo waxay isu heleen macluumaad badan intii ay u baahnaayeen inay wax ka qabtaan su'aalahooda cilmi-baarista. Halkii ayan isticmaalin xogta si buuxda, waxay ku baran lahaayeen xajmiga saameynta sida shaqeynta dareenka dareen-celinta kahor. Sida Schultz et al. (2007) ogaatay in saameynta daaweynta ay ka duwan tahay dadka isticmaala iftiinka iyo culus, laga yaabo inay saameyn ku yeeshaan wargeyska Feed Feed inay ka duwan yihiin dadka horey u soo gudbiyay farriimo farxad (ama murugo). Dib-u-celinta waxay keeni kartaa "kalluumeysiga" (Humphreys, Sierra, and Windt 2013) iyo "Hack-hacking" (Simmons, Nelson, and Simonsohn 2011) , laakiin kuwani waxa ay si weyn u wajahayaan warbixinno daacad ah (Simmons, Nelson, and Simonsohn 2011) , horey u diiwaangelinta (Humphreys, Sierra, and Windt 2013) , iyo hababka barashada mashiinka isku dayaya in ay iska ilaaliyaan khaladka.