Yini okufanele ufunde ngokulandelayo

• Isingeniso (isigaba 4.1)

Imibuzo ngokuphathelene nokucwaninga kwezenhlalakahle ngokuvamile iyanzima futhi iyinkimbinkimbi. Ukuze uthole indlela yokusekela okuyisisekelo esekelwe emagrafu ye-causal, bheka i- Pearl (2009) , nangendlela yokwenza isisekelo esekelwe kwimiphumela engase ibe khona, bheka u- Imbens and Rubin (2015) . Ukuze uqhathanise phakathi kwalezi zindlela ezimbili, bheka Morgan and Winship (2014) . Ukuze uthole indlela ehlelekile yokuchaza ukudideka, bheka u- VanderWeele and Shpitser (2013) .

Kulesi sahluko, ngidalile ukuthi kubonakala sengathi kuluhlaza phakathi kokukwazi kwethu ukwenza izilinganiso ze-causal kusuka kwedatha yokuhlola nokungahlolisi. Kodwa-ke, ngicabanga ukuthi, empeleni, umehluko ukhululekile kakhulu. Isibonelo, wonke umuntu uyavuma ukuthi ukubhema kubangelwa umdlavuza, yize kungenjalo ukuhlolwa okulawulwa ngokungahleliwe okuphoqa abantu ukuthi bashunqise kwenzeke. Ukuze uthole ukwelashwa Shadish, Cook, and Campbell (2001) yokuhlola ubone i- Rosenbaum (2002) , ( ??? ) , Shadish, Cook, and Campbell (2001) , Dunning (2012) .

Izigaba 1 no-2 ze- Freedman, Pisani, and Purves (2007) zinikeza isingeniso esicacile sokungafani phakathi kokuhlolwa, ukuhlolwa okulawulwayo, nokuhlolwa okulawulwa ngokungahleliwe.

Manzi (2012) inikeza isingeniso esithakazelisayo nesifundwayo ngaphansi kwezingqungquthela zefilosofi nezibalo zokuhlolwa okulawulwa ngokungahleliwe. Ibuye inikeze izibonelo ezithakazelisayo zomhlaba wangempela wamandla okuhlola ebhizinisini. Issenberg (2012) inikeza isingeniso esithakazelisayo ekusetshenzisweni kokuhlolwa emikhankaso yezombangazwe.

• Yiziphi izivivinyo? (isigaba 4.2)

Box, Hunter, and Hunter (2005) , @ casella_statistical_2008, no- Athey and Imbens (2016b) banikeza izethulo ezinhle Athey and Imbens (2016b) zokuhlela nokuhlaziywa. Ngaphezu kwalokho, kunezokwelapha ezinhle kakhulu zokusetshenziswa kwezivivinyo ezindaweni eziningi ezihlukene: ezomnotho (Bardsley et al. 2009) , ezomphakathi (Willer and Walker 2007; Jackson and Cox 2013) , i-psychology (Aronson et al. 1989) , isayensi yezombusazwe (Morton and Williams 2010) , kanye nenqubomgomo yomphakathi (Glennerster and Takavarasha 2013) .

Ukubaluleka kokuqashwa kwabahlanganyeli (isb., Isampuli) kuvame ukuhlolisiswa ekucwaningweni kokuhlola. Kodwa-ke, uma umphumela wokwelapha uyingqayizivele kubantu, isampuli siyadingeka. Longford (1999) yenza leli phuzu ngokucacile lapho ekhuthaza abacwaningi bacabange ngezivivinyo njengenhlolovo yabantu ngesampuli esingenasici.

• Ubukhulu obubili bokuhlolwa: ibanga-le-analog-digital (isigaba 4.3)

Ngiphakamisile ukuthi kukhona ukuqhubeka phakathi kokuhlolwa kwelebhu namasimu, kanti abanye abacwaningi bahlongoze izintambo eziningi, ikakhulukazi ezihlukanisa izinhlobo ezahlukene zokuhlolwa kwensimu (Harrison and List 2004; Charness, Gneezy, and Kuhn 2013) .

Amaphepha ambalwa aqhathanise ama-laboratory kanye nokuhlolwa kwamasimu ku-abstract (Falk and Heckman 2009; Cialdini 2009) futhi ngokwemiphumela yezilinganiso ezithile ezenzweni zesayensi yezombusazwe (Coppock and Green 2015) , ukonga (Levitt and List 2007a, 2007b; Camerer 2011; Al-Ubaydli and List 2013) , kanye ne-psychology (Mitchell 2012) . Jerit, Barabas, and Clifford (2013) banikeza umklamo omuhle wokucwaninga ngokuqhathanisa imiphumela Jerit, Barabas, and Clifford (2013) namasimu. Parigi, Santana, and Cook (2017) ichaza ukuthi ukuhlolwa kwenkambu ye-intanethi ingahlanganisa kanjani ezinye zezici zokuhlolwa kwelebhu namasimu.

Ukukhathazeka mayelana nabahlanganyeli bashintsha ukuziphatha kwabo ngoba bayazi ukuthi babhekwa ngokucophelela ngezinye izikhathi kuthiwa yimiphumela yokufuna , futhi baye bafundiswa kwengqondo (Orne 1962) kanye (Zizzo 2010) . Nakuba iningi lihlotshaniswa nokuhlolwa kwebhu, lezi zinkinga ezifanayo zingabangela izinkinga zokuhlolwa kwensimu kanye. Eqinisweni, imiphumela yokufuna izikhathi ngezinye izikhathi ibizwa ngokuthi imiphumela ye-Hawthorne , igama elithola ukuhlolwa okudumile okukhanyisa okwaqala ngo-1924 kwi-Hawthorne Works ye-Western Electric Company (Adair 1984; Levitt and List 2011) . Zombili zifuna imiphumela kanye nemiphumela ye- Hawthorne ehlobene eduze nomqondo wokulinganisa okusebenzayo okukhulunywa ngakho esahlukweni 2 (bheka futhi Webb et al. (1966) ).

Ukuhlolwa kwamasimu kunomlando omude wezomnotho (Levitt and List 2009) , isayensi yezombangazwe (Green and Gerber 2003; Druckman et al. 2006; Druckman and Lupia 2012) , psychology (Shadish 2002) , kanye nenqubomgomo yomphakathi (Shadish and Cook 2009) . Indawo eyodwa yesayensi yezenhlalakahle lapho ukuhlolwa kwensimu ngokushesha kuvelele ukuthuthukiswa kwamazwe ngamazwe. Ukubukeza okuhle kwalowo msebenzi ngaphakathi kwezomnotho ubone Banerjee and Duflo (2009) , futhi ngokuhlolwa okubucayi bheka Deaton (2010) . Ukubukeza lo msebenzi ezenzweni zesayensi yezombusazwe ubone Humphreys and Weinstein (2009) . Okokugcina, izinselele zokuziphatha ezivela ezihlolweni zasensimini ziye zahlolwa ngokwesimo sezesayensi yezombusazwe (Humphreys 2015; Desposato 2016b) nokuthuthukiswa kocwaningo (Baele 2013) .

Kulesi sigaba, ngiphakamise ukuthi ulwazi lwangaphambi kokwelapha lungasetshenziselwa ukuthuthukisa ukucacisa kwemiphumela yokwelashwa okulinganisiwe, kodwa kukhona impikiswano mayelana nale ndlela; bheka Freedman (2008) , W. Lin (2013) , Berk et al. (2013) , no- Bloniarz et al. (2016) ngolwazi oluthe xaxa.

Okokugcina, kunezinye izinhlobo ezimbili zokuhlolwa okwenziwe ososayensi bezenhlalakahle ezingahambelani kahle ngokubaluleka kwebhu lab-field: ukuhlolwa kocwaningo kanye nokuhlolwa kwezenhlalakahle. Uhlolo lokuhlola luyizingqinamba ezisebenzisa ingqalasizinda yezinhlolovo ezikhona futhi uqhathanise nezimpendulo kwezinye izinguqulo zemibuzo efanayo (okunye okuhlolwa kohlolo kuvezwe eSahlukweni 3); ukuze uthole okwengeziwe kokuhlolwa kocwaningo bheka Mutz (2011) . Ukuhlolwa kwezenhlalakahle yimizamo lapho ukwelashwa kungumgomo othile wezenhlalakahle ongasetshenziswa kuphela nguhulumeni. Ukuhlolwa kwezenhlalakahle kuhlobene kakhulu nokuhlolwa kohlelo. Ukuze uthole okwengeziwe mayelana nokuhlolwa kwenqubomgomo, bheka Heckman and Smith (1995) , i- Orr (1998) , no-glennerster_running_2013.

• Ukuhamba ngaphandle kokuhlolwa okulula (isigaba 4.4)

Ngikhethe ukugxila ezintweni ezintathu: ukuqinisekiswa, ukuphazamiseka kwemithi yokwelashwa nemiphumela. Lezi zincazelo zinamagama ahlukene emasimini ahlukene. Isibonelo, izazi zezokwelapha zivame ukuhambisa ngaphezu kokuhlolwa okulula ngokugxila kubaxhumanisi nabomengameli (Baron and Kenny 1986) . Umqondo wabaxhumanisi uthathwe yilokho engikubiza ngakho, futhi umqondo wabamengameli uthathwe yilokho engikubiza ngokusebenza kwangaphandle (isb., Ngabe imiphumela yokuhlolwa ingafani uma iqhutshwa ezimweni ezihlukene) kanye nokuphazamiseka kwemiphumela yokwelashwa ( isib, imiphumela ikhudlwana kwabanye abantu kunabanye).

Ukuhlolwa Schultz et al. (2007) ibonisa ukuthi imibono yezenhlalo ingasetshenziswa kanjani ukudala ukungenelela okuphumelelayo. Ukuze uthole ingxabano evamile mayelana nendima yombono ekudaleni ukungenelela okuphumelelayo, bheka i- Walton (2014) .

• Ukuqinisekiswa (isigaba 4.4.1)

Imiqondo yokuqinisekisa kwangaphakathi nangaphandle yangeniswa kuqala Campbell (1957) . Bheka Shadish, Cook, and Campbell (2001) ukuze uthole umlando oningiliziwe kanye nokuqaphela ngokucophelela isiphetho sesibalo, ukuqinisekiswa kwangaphakathi, ukwakha ukuqinisekiswa, nokusebenza kwangaphandle.

Ukuze uthole ukubuyekezwa kwezindaba ezihlobene nesiphetho sesibalo esivumelekile ekuhlolweni bheka i- Gerber and Green (2012) (kusukela ekuboneni kwesayensi yezenhlalakahle) no- Imbens and Rubin (2015) (kusuka kumbono wezibalo). Ezinye izimpikiswano zesibalo sezibalo ezivela ngqo ekuhloleni kwenkambu ye-intanethi kufaka phakathi izinkinga ezifana nezindlela eziphumelelayo zokudala (Bakshy and Eckles 2013) nedatha esekelwe (Bakshy and Eckles 2013) .

Ukusebenza kwangaphakathi kungaba nzima ukuqiniseka ezinkingeni zensimu eziyinkimbinkimbi. Bheka, isibonelo, i- Gerber and Green (2000) , i- Imai (2005) , no- Gerber and Green (2005) ye-mpikiswano mayelana nokuqaliswa kokuhlolwa kwensimu eyinkimbinkimbi mayelana nokuvota. Kohavi et al. (2012) Kohavi et al. (2013) inikezela isingeniso ezinselele zokuqinisekisa ukuhamba kwesikhathi ezindaweni zokuhlola inthanethi.

Enye ingozi enkulu ekuqinisekiseni kwangaphakathi kungenzeka ukuthi i-randomization ehlulekile. Enye indlela engaba khona yokuthola izinkinga nge-randomization ukuqhathanisa amaqembu wokwelapha kanye nokulawula ezinkambisweni ezibonakalayo. Lolu hlobo lokuqhathaniswa lubizwa ngokuthi isheke lokulinganisela . Bheka Hansen and Bowers (2008) ngenqubo yokuhlola ukulinganisela nokuhlola i- Mutz and Pemantle (2015) ngokukhathazeka mayelana nokuhlola kwe-balance. Isibonelo, usebenzisa isheke lokulinganisela, u- Allcott (2011) uthole ubufakazi bokuthi ukungahleliwe akuzange kwenziwe ngendlela efanele ku-Three of Test experiments (bheka ithebula 2; amasayithi 2, 6, no-8). Ngezinye izindlela, bheka isahluko 21 sika- Imbens and Rubin (2015) .

Ezinye izikhathazo ezinkulu ezihlobene nokusebenza kwangaphakathi ziyizi: (1) ukungahambisani okuhlangene okukodwa, lapho kungabikho wonke umuntu eklinikweni yokwelapha okwenziwe ukwelashwa, (2) ukungabambisani okuhlangene, lapho kungabikho wonke umuntu eklasini yokwelashwa ethola ukwelashwa kanye nabanye abantu iqembu lokulawula lithola ukwelashwa, (3) umzila, lapho imiphumela engalinganiswanga khona kwabanye ababambiqhaza, kanye (4) nokuphazanyiswa, lapho ukwelashwa kuchitheka khona kubantu abasesimweni sokwelashwa kubantu abakhona esimweni sokulawula. Bheka izahluko 5, 6, 7, no-8 ze- Gerber and Green (2012) ukuze uthole okuningi kulezi zinkinga.

Ukuze uthole okwengeziwe ekuwakheni ukufaneleka, bheka i- Westen and Rosenthal (2003) , nokunye okwengeziwe ekwakheni ukufaneleka kwemithombo emikhulu yedatha, i- Lazer (2015) nesahluko 2 sale ncwadi.

Esinye isici sokusebenza kwangaphandle yisimiso lapho ukungenelela kuhlolwa khona. Allcott (2015) inikeza ukwelashwa ngokucophelela nokuqondisa kokukhethwa kwendawo yokukhetha indawo. Le nkinga iphinde ixoxwe Deaton (2010) . Esinye isici sokusebenza kwangaphandle ukuthi ngabe ukusebenza okuhlukile kokungenelela okufanayo kuzoba nemiphumela efanayo. Kulokhu, ukufaniswa phakathi Schultz et al. (2007) no- Allcott (2011) kubonisa ukuthi ukuhlolwa kwe-Opower kunomthelela ophansi wokuphathwa okwedlula ukwenziwa kokuqala kuka-Schultz nozakwabo (1.7% kuya kwe-5%). Allcott (2011) ucabange ukuthi ukuhlolwa okulandelwayo kwaba nomthelela omncane ngenxa yezindlela zokwelapha ezahluke ngayo: i-emoticon ngesandla ebhaliwe njengengxenye yocwaningo elixhaswe yinyuvesi, uma kuqhathaniswa ne-emoticon ephrintiwe njengengxenye ye-mass-produced umbiko kusuka enkampanini yamandla.

• Ukuhlushwa kwemithi yokwelapha (isigaba 4.4.2)

Ukuze uthole umbono obala kakhulu we-heterogeneity yemiphumela yokwelapha emayilini yokuhlola, bheka isahluko 12 se- Gerber and Green (2012) . Ukuze uthole izethulo zokungenwa kwemithi yokwelashwa ekuhlolweni kwezokwelapha, bheka Kent and Hayward (2007) , Longford (1999) , Kravitz, Duan, and Braslow (2004) . Ukucatshangelwa kwe-hterogeneity yemiphumela yokwelashwa ngokuvamile kugxile ekuhlukeni okususelwe kwizici zangaphambi kokwelashwa. Uma unesithakazelo ku-heterogeneity ngokusekelwe emiphumeleni yokuthunyelwa kwezokuthutha, kuyadingeka izindlela eziyinkimbinkimbi, njenge-stratification eyinhloko (Frangakis and Rubin 2002) ; bheka Page et al. (2015) ukuze kubuyekezwe.

Abacwaningi abaningi bacatshangisa ukuthi ukuphathwa kwemithi yokwelashwa okungafani nokusebenzisa ukuguqulwa komzimba, kodwa izindlela ezintsha zithembele ekufundeni komshini; bheka, isibonelo, Green and Kern (2012) , u- Imai and Ratkovic (2013) , Taddy et al. (2016) , no- Athey and Imbens (2016a) .

Kukhona ukungabaza mayelana nokutholakala kwe-heterogeneity yemiphumela ngenxa yezinkinga eziningi zokuqhathanisa kanye "nokudoba." Kunezinhlobonhlobo zezindlela zezibalo ezingasiza ukubhekana nokukhathazeka ngokuqhathanisa okuningi (Fink, McConnell, and Vollmer 2014; List, Shaikh, and Xu 2016) . Enye indlela yokukhathazeka mayelana "nokudoba" i-pre-registration, okuyinto iyaba yinto evame kakhulu kwi-psychology (Nosek and Lakens 2014) , isayensi yezombangazwe (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , kanye nezomnotho (Olken 2015) .

Esifundweni Costa and Kahn (2013) cishe ingxenye yesigamu semindeni ekuhlolweni kungaxhunyaniswa nolwazi lwabantu. Abafundi abanesithakazelo kulemininingwane kufanele babheke iphepha lokuqala.

• Izindlela (isigaba 4.4.3)

Izinqubo zibaluleke kakhulu, kodwa ziba nzima kakhulu ukutadisha. Ucwaningo mayelana nezinqubo luhlobene eduze nokucwaninga kwabaxhumanisi kwezokwelapha (kodwa bona futhi VanderWeele (2009) ukuze uqhathanise ngokucacile phakathi kwemibono emibili). Izindlela zesitatimende zokuthola izindlela, njengendlela eyenziwe Baron and Kenny (1986) , zivamile. Ngeshwa, kuvela ukuthi lezo zinqubo zixhomeke ekucabangeni okuqinile (Bullock, Green, and Ha 2010) futhi bahlupheka uma kunezinqubo eziningi, njengoba umuntu angalindela ezimweni eziningi (Imai and Yamamoto 2013; VanderWeele and Vansteelandt 2014) . Imai et al. (2011) no- Imai and Yamamoto (2013) banikeze izindlela ezithuthukisiwe zezibalo. Ngaphezu kwalokho, i- VanderWeele (2015) inikeza ukwelashwa ubude bezincwadi ngemiphumela eminingi ebalulekile, kuhlanganise nendlela ephelele yokuhlaziywa kobuzwe.

Indlela ehlukile igxile ekuhlolweni okuzama ukusebenzisa indlela ngokuqondile (isib., Ukunikeza abahamba nge-vitamin C). Ngeshwa, ezimisweni eziningi zesayensi yezenhlalakahle, ngokuvamile zikhona izindlela eziningi futhi kunzima ukuklama imithi eguqule eyodwa ngaphandle kokushintsha abanye. Ezinye izindlela ezisetshenziselwa ukushintsha izindlela ezichazwayo zichazwe ngu- Imai, Tingley, and Yamamoto (2013) , Ludwig, Kling, and Mullainathan (2011) , kanye Pirlott and MacKinnon (2016) .

Abacwaningi abaqhuba ukuhlola okusemthethweni okudingekayo kuzodingeka bakhathazeke ngokuhlolwa okuningi kwe-hypothesis; bheka Fink, McConnell, and Vollmer (2014) List, Shaikh, and Xu (2016) ukuthola ulwazi oluthe xaxa.

Okokugcina, izindlela zibuye zibe nemlando ende kufilosofi yesayensi njengoba kuchazwe Hedström and Ylikoski (2010) .

• Ukusebenzisa izindawo ezikhona (isigaba 4.5.1)

Ukuze uthole olunye ulwazi mayelana nokusetshenziswa kwezifundo zokuxhumana kanye nokuhlolwa kwamabhuku ukuhlola ukucwaswa, bheka Pager (2007) .

• Yakha ukuhlolwa kwakho (isigaba 4.5.2)

Indlela evamile kakhulu yokubamba iqhaza ekuhloliseni ukuthi uzakhela i-Amazon Mechanical Turk (MTurk). Ngenxa yokuthi i-MTurk ihlaziya izici zemizamo yendabuko-labakhokhelwayo yokuqedela imisebenzi abangayikuyenza kubaphenyi abakhululekile-basebenzile kakade basebenzisa ama-Turkers (abasebenzi ku-MTurk) njengabahlanganyeli bokuhlola, okubangela ukuqoqwa kwedatha okusheshayo nangabizi kangako okungaphumelela ekuhlolweni kwe-laboratory (Paolacci, Chandler, and Ipeirotis 2010; Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012; Rand 2012; Berinsky, Huber, and Lenz 2012) .

Ngokuvamile, izinzuzo ezinkulu kakhulu zokusebenzisa abahlanganyeli abaqashiwe kusuka ku-MTurk ziyizinsiza. Nakuba ukuhlolwa kwelabhu kungathatha amasonto ukuqhuba nokuhlolwa kwensimu kungathatha izinyanga ukusetha, ukuhlolwa kwabahlanganyeli abaqashiwe kusuka ku-MTurk kungenziwa ngezinsuku. Isibonelo, Berinsky, Huber, and Lenz (2012) bakwazi ukuzuza izifundo ezingu-400 ngolunye usuku ukuze bahlanganyele ekuhlolweni okungamaminithi angu-8. Ngaphezu kwalokho, laba bahlanganyeli bangabhaliswa cishe nganoma iyiphi inhloso (kubandakanya ukuhlola nokubambisana okukhulu, njengoba kuxoxwa ngazo ezahlukweni 3 no-5). Lokhu kulula ukuqasha kusho ukuthi abacwaningi bangasebenzisa ukulandelana kwezivivinyo ezihlobene ngokulandelana okusheshayo.

Ngaphambi kokuqasha abahlanganyeli abavela ku-MTurk ngezivivinyo zakho, kunezinto ezine ezibalulekile okudingeka wazi. Okokuqala, abacwaningi abaningi banokungabaza okungafani kokuhlolwa okubandakanya iTurkers. Ngenxa yokuthi lokhu kungabazeki, akunzima ukuphikisana nobufakazi. Kodwa-ke, ngemva kweminyaka eminingi yokufunda esebenzisa abaTurkers, manje singaphetha ngokuthi lokhu kungabaza ngokungafanelekile. Kube nokucwaninga okuningi okuqhathaniswa nokwaziswa kwabantu baseTurkers nalabo abanye abantu kanye nezifundo eziningi eziqhathanisa imiphumela yokuhlolwa no-Turkers njengabo abavela kwabanye abantu. Njengoba sinikezwe wonke lo msebenzi, ngicabanga ukuthi indlela engcono kakhulu yokucabanga ngayo ukuthi abaTurkers bayisampula esivumelana kahle, kufana nabafundi kodwa abahlukahlukene kakhulu (Berinsky, Huber, and Lenz 2012) . Ngakho-ke, njengoba nje abafundi bengabantu abanengqondo kwabanye, kodwa hhayi bonke, ucwaninga, abaTurkers bangabantu abanengqondo kwabanye, kodwa hhayi bonke, ucwaningo. Uma uzosebenza noTurkers, kuwukuhlakanipha ukufunda eziningi zezifundo eziqhathaniswayo nokuqonda imiqondo yazo.

Okwesibili, abacwaningi baye bahlakulela imikhuba engcono yokwandisa ukuhlolwa kwangaphakathi kwe-MTurk, futhi kufanele ufunde futhi ulandele lezi zindlela ezinhle kakhulu (Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012) . Isibonelo, abacwaningi abasebenzisa i-Turkers bayakhuthazwa ukuba basebenzise ama-screeners ukususa abahlanganyeli (Berinsky, Margolis, and Sances 2014, 2016) (kodwa bona no- DJ Hauser and Schwarz (2015b) no- DJ Hauser and Schwarz (2015a) ). Uma ungasusi abahlanganyeli abangenakulingisa, noma yikuphi umphumela wokwelashwa kungahlanzwa ngomsindo abawuthulayo, futhi ngokusemthethweni inani labahlanganyeli abangalindelekile lingaba lukhulu. Ekuhlolweni kukaHuber nozakwabo (2012) , abangaba ngu-30% wabahlanganyeli abahlulekile ukunakekelwa kwezicabangela eziyisisekelo. Ezinye izinkinga ezivame ukuvela lapho abantu baseTurkers besetshenziselwa abahlanganyeli abangenabanikazi (Chandler et al. 2015) kanye nomzila (Zhou and Fishbach 2016) .

Okwesibili, ngokuphathelene nezinye izinhlobo zokuhlolwa kwedijithali, ukuhlolwa kwe-MTurk akukwazi ukukala; Stewart et al. (2015) kulinganisa ukuthi kunoma yisiphi isikhathi esithile kunabantu abangaba ngu-7 000 kuphela ku-MTurk.

Okokugcina, kufanele wazi ukuthi uMaturk ungumphakathi onemithetho yawo kanye nemigomo (Mason and Suri 2012) . Ngendlela efanayo ukuthi uzama ukuthola ngayo isiko lezwe lapho uzobe uqhuba khona izivivinyo zakho, kufanele uzame ukuthola okwengeziwe mayelana namasiko kanye nemigomo yeTurkers (Salehi et al. 2015) . Futhi kufanele wazi ukuthi abaTurkers bazobe bekhuluma ngokuzama kwakho uma wenza okuthile okungalungile noma okungahambi kahle (Gray et al. 2016) .

I-MTurk iyindlela elula kakhulu yokubamba iqhaza ekuhloliseni kwakho, kungakhathaliseki ukuthi i-lab-like, njengekaHuber Huber, Hill, and Lenz (2012) , noma okunye okufana nendawo, njengekaMason Mason and Watts (2009) , Goldstein, McAfee, and Suri (2013) , i- Goldstein et al. (2014) , Horton and Zeckhauser (2016) , Mao et al. (2016) .

• Yakha umkhiqizo wakho (isigaba 4.5.3)

Uma ucabanga ngokuzama ukwenza umkhiqizo wakho, ngincoma ukuthi ufunde iseluleko esinikezwa iqembu le-MovieLens Harper and Konstan (2015) . Ukuqondisisa okuyisisekelo okuvela kulokho okuhlangenwe nakho kwabo ukuthi kwiprojekthi ngayinye ephumelelayo kunamaphuzu amaningi, amaningi. Isibonelo, iqembu le-MovieLens lenye iminye imikhiqizo, njengeGopherAnswers, eyayihluleka ngokuphelele (Harper and Konstan 2015) . Esinye isibonelo somcwaningi sihluleka ngenkathi ezama ukwenza umkhiqizo umzamo ka-Edward Castronova wokwakha umdlalo we-inthanethi obizwa nge-Arden. Naphezu kwamaRandi ayizigidi ezingama-250,000, lo msebenzi wawuyi-flop (Baker 2008) . Amaphrojekthi afana neGopherAnswers ne-Arden ngeshwa kakhulu kakhulu kunamaphrojekthi afana ne-MovieLens.

• Uzakwethu onamandla (isigaba 4.5.4)

Ngizwile umbono we-Quadrant kaPasteur okukhulunywe ngazo njalo ezinkampanini zezobuchwepheshe, futhi kusiza ukuhlela imizamo yocwaningo e-Google (Spector, Norvig, and Petrov 2012) .

Ucwaningo lwama-Bond kanye nozakwethu (2012) luzama futhi ukuthola imiphumela yalezi zokwelashwa kubangane balabo ababazitholile. Ngenxa yokwakhiwa kokuhlolwa, lezi zinhlayiya kunzima ukubona ukuthi zihlanzekile; abafundi abanentshisekelo kufanele babone Bond et al. (2012) ukuze uthole ingxoxo egcwele. UJones kanye nosebenza nabo (2017) nabo baqhuba ukuhlolwa okufanayo ngesikhathi samakhetho ka-2012. Lezi zivivinyo ziyingxenye yenkambiso ende yokuhlolwa kwesayensi yezombangazwe emizamweni yokukhuthaza ukuvota (Green and Gerber 2015) . Lezi zivivinyo eziphuma ngaphandle kokuvota zivamile, ngokwengxenye ngoba zisemkhawulweni wePasteur. Okungukuthi, kunabantu abaningi abagqugquzelwa ukwandisa ukuvota nokuvota kungaba ukuziphatha okuthakazelisayo ukuhlola imibono eminingi jikelele mayelana nokuguquka kokuziphatha kanye nomthelela wezenhlalo.

Ukuze uthole iseluleko mayelana nokusebenza kokuhlolwa kwensimu nezinhlangano zomlingani ezifana namaqembu ezombangazwe, ama-NGO, namabhizinisi, bheka Loewen, Rubenson, and Wantchekon (2010) , i- JA List (2011) , Gueron (2002) . Ukuze uthole imibono mayelana nokuthi ukusebenzisana nezinhlangano kungathinta kanjani imiklamo yocwaningo, bheka i- King et al. (2007) Green, Calfano, and Aronow (2014) . Ukubambisana kungabangela nemibandela yokuziphatha, njengoba kuxoxwa yi- Humphreys (2015) Nickerson and Hyde (2016) .

• Iseluleko sokwakha (isigaba 4.6)

Uma udala uhlelo lokuhlaziya ngaphambi kokusebenzisa ukuhlolwa kwakho, ngiphakamisa ukuthi uqale ngokufunda imihlahlandlela yokubika. I-CONSORT (Ukuhlanganiswa Kwemigomo Yemibiko Yesivivinyo) yenziwe ngemithi (Schulz et al. 2010) futhi iguqulelwe ukucwaninga kwezenhlalo (Mayo-Wilson et al. 2013) . Isitatimende esilandelayo semihlahlandlela sakhiwe ngabahleli be- Journal of Experimental Political Science (Gerber et al. 2014) (bheka futhi Mutz and Pemantle (2015) Gerber et al. (2015) ). Ekugcineni, imihlahlandlela yokubika iye yasungulwa kwengqondo (APA Working Group 2008) , futhi ubone Simmons, Nelson, and Simonsohn (2011) .

Uma udala uhlelo lokuhlaziya, kufanele ucabange ngaphambi kokubhalisa ngoba ukubhalisa kwangaphambi kwesikhathi kuzokwandisa ukuzethemba abanye abakwenzayo emiphumeleni yakho. Ngaphezu kwalokho, uma usebenza nomlingani wakho, kuzomkhawuloza ikhono lomlingani wakho lokushintsha ukuhlaziywa ngemuva kokubona imiphumela. Ukubhalisa kwangaphambi kokubhalisa sekuya ngokuya ngokuvame kakhulu kwengqondo (Nosek and Lakens 2014) , isayensi yezombusazwe (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , kanye (Olken 2015) .

Iseluleko sokuklama ngokukhethekile ekuhlolweni kwendawo Konstan and Chen (2007) kukhonjiswe ku- Konstan and Chen (2007) Chen and Konstan (2015) .

Lokho engikubiza ngokuthi isu le-armada ngezinye izikhathi kuthiwa ucwaningo lwenqubo ; bheka Wilson, Aronson, and Carlsmith (2010) .

• Dala idatha yezindleko eziguquguqukayo (isigaba 4.6.1)

Ukuze uthole okwengeziwe kokuhlolwa kwe- Salganik, Dodds, and Watts (2006) , bheka i- Salganik, Dodds, and Watts (2006) , i- Salganik and Watts (2008) , i- Salganik and Watts (2009b) , i- Salganik and Watts (2009a) , ne- Salganik (2007) . Ukuze uthole okwengeziwe ezimakethe zokuwina, qaphela Frank and Cook (1996) . Ukuze uthole okwengeziwe mayelana nokukhishwa inhlanhla namakhono ngokwengeziwe, bheka Mauboussin (2012) , Watts (2012) , Frank (2016) .

Kukhona enye indlela yokuqeda izinkokhelo ezibandakanyekayo abacwaningi okufanele bazisebenzise ngokuqapha: ukubhalisa. Eziningi zokuhlola insimu ye-intanethi ngokuyinhloko zibhalwa ekuhlolweni futhi azizange zihlawuliswe. Izibonelo zalolu daba zibandakanya u-Restivo no-van de Rijt's (2012) ukuhlolwa kwemivuzo e-Wikipedia naseBond kanye no-colleague's (2012) ukuzama ukukhuthaza abantu ukuba bavote. Lezi zivivinyo azinazo izindleko eziguquguqukayo-kunalokho, zinezindleko eziguquguqukayo eziguquguqukayo kubacwaningi . Kulezi zivivinyo, ngisho noma izindleko zomhlanganyeli ngamunye zincane kakhulu, izindleko eziningi zingaba zikhulu kakhulu. Abacwaningi abaqhuba ukuhlolwa okukhulu kwe-intanethi kuvame ukufakazela ukubaluleka kwemiphumela encane yokwelashwa okubalinganiselwayo ngokuthi lemiphumela emincane ingaba ibalulekile uma isetshenziswa kubantu abaningi. Ukucabanga okufanayo okusebenzayo kubhekene nezindleko abacwaningi ababeka kubathintekayo. Uma ukuhlolwa kwakho kubangela abantu abayizigidi ezilodwa ukuba bachithe umzuzu owodwa, ukuhlolwa akulimazi kunoma yimuphi umuntu, kodwa ngokubanzi sekulahlekelwe cishe iminyaka emibili.

Enye indlela yokudala ukukhokhelwa kwezindleko eziguquguqukayo kubahlanganyeli ukusebenzisa i-lottery, indlela esetshenziselwe nokucwaninga kocwaningo (Halpern et al. 2011) . Ukuze uthole okwengeziwe mayelana nokuklama okuhlangenwe nakho okujabulisayo komsebenzisi, bheka Toomim et al. (2011) . Ukuze uthole okwengeziwe mayelana nokusebenzisa ibhu ukudala ukuhlolwa kwezindleko eziguquguqukayo bheka ( ??? ) .

• Faka esikhundleni, Hlunga, futhi unciphise (isigaba 4.6.2)

Ama-R amathathu njengoba ahlongozwa Russell and Burch (1959) anjengoba:

"Ukubuyiselwa kushiwo esikhundleni abazi ophilayo izilwane ephakeme material insentient. Ukuncishiswa kusho ukuncipha izinombolo izilwane esetshenziswa ukuthola ulwazi of imali inikezwe nokunemba. Intuthuko kushiwo noma yimuphi ukwehla izigameko noma kobukhulu izinqubo ngonya nakilabo izilwane kusadingeka ukuba kusetshenziswe. "

Ama-R amathathu engiwaphakamisayo awafaki ngokweqile izimiso zokuziphatha ezichazwe esahlukweni 6. Kunalokho, yiyona inguqulo echazwe kabanzi enye yalezo zimiso-inzuzo-ngokukhethekile ekusetshenzisweni kokuhlolwa kwabantu.

Ngokwe-R yokuqala ("ukushintshwa"), uma kuqhathaniswa nokuzamazama kokucindezeleka ngokomzwelo (Kramer, Guillory, and Hancock 2014) kanye nokuzamazama kwemvelo ngokomzwelo (Lorenzo Coviello et al. 2014) inikeza izifundo eziningi mayelana nokuhweba okuhilelekile ekuhambeni kusuka ekuhlolweni kuya ekuhlolweni kwemvelo (nezinye izindlela ezifana nokulinganisa lokho okuzama ukulinganisa okulinganiselwe kwedatha engekho yokuhlola; bheka isahluko 2). Ngaphandle kwezinzuzo zokuziphatha, ukusukela ekuhlolweni kuya kwizifundo ezingekho kokuhlola kwenza abacwaningi bakwazi ukuhlola ukwelashwa abakwazi ukukwenza. Lezi zinzuzo zokuziphatha nezokungena ziza ngezindleko, noma kunjalo. Ngokuhlolwa kwemvelo abacwaningi banokulawulwa okuncane ezintweni ezifana nokuqashwa kwabahlanganyeli, ukungahleliwe, kanye nendlela yokwelapha. Isibonelo, umkhawulo owodwa wemvula njengendlela yokwelapha ukuthi kokubili ukwandisa ubuhle futhi kunciphise ukungabi nalutho. Esifundweni sokuhlola, noma kunjalo, i-Kramer kanye nozakwethu bakwazi ukulungisa ukubekezela nokungahambisani nokuzimela ngokuzimela. Indlela ethile esetshenziswa Lorenzo Coviello et al. (2014) laphinde lachazwa L. Coviello, Fowler, and Franceschetti (2014) . Ukuze uthole isingeniso ezinhlobonhlobo zezinsimbi, okuyinto indlela esetshenziswa ngayo Lorenzo Coviello et al. (2014) , bheka i- Angrist and Pischke (2009) ( Angrist, Imbens, and Rubin (1996) kakhulu) noma i- Angrist, Imbens, and Rubin (1996) (ehlelekile kakhulu). Ukuhlola okungaqondakali kwezinguquko zezinsimbi, bheka Deaton (2010) , kanye nokwethulwa kokuguquguquka kwezinsimbi nezinsimbi ezibuthakathaka (imvula iyisitsha esibuthakathaka), bheka u- Murray (2006) . Ngokuvamile, isingeniso esihle semilingo yemvelo sinikezwa Dunning (2012) , kuyilapho Rosenbaum (2002) , ( ??? ) , Shadish, Cook, and Campbell (2001) inikeza imibono emihle mayelana nokulinganisa imiphumela ye-causal ngaphandle kokuhlola.

Ngokwesibili R ("ukulungiswa"), kukhona ukuhweba kwezesayensi nezokungena lapho kucatshangelwa ukuguqula umklamo wokuThuthukiswa koMzwelo ukuvimbela okuthunyelwe ukukhulisa okuthunyelwe. Isibonelo, kungenzeka ukuthi ukuqaliswa kobuchwepheshe be-News Feed kwenza kube lula ukwenza okuvivinywa ukuthi yiziphi izithuba ezivinjiwe kunokuba zikhuthazwe (phawula ukuthi ukuhlolwa okubandakanya ukuvimba ukuthunyelwe kungaqaliswa njengendlela ungqimba phezu kwesistimu ye-News Feed ngaphandle kwesidingo sokushintsha kwesistimu engaphansi). Ngokwesayensi, noma kunjalo, inkolelo eqondiswe ukuhlolwa ayizange icacise ngokucacile ukuklama okukodwa kunomunye. Ngeshwa, angikwazi ukucwaninga okuphambili kokuphathelene nokufaneleka okuhambisana nokuvimbela nokwandisa okuqukethwe ku-News Feed. Futhi, angizange ngibone ucwaningo oluningi mayelana nokuhlanza ukwelashwa ukubenza bangabi nengozi kakhulu; Okunye okuhlukile ku- B. Jones and Feamster (2015) , okucabangela icala lokulinganiswa kwe-Inthanethi (isihloko engikhuluma ngaso esahlukweni 6 ngokuhambisana nesifundo se-Encore (Burnett and Feamster 2015; Narayanan and Zevenbergen 2015) ).

Ngokwesithathu R ("ukunciphisa"), izethulo ezinhle zokuhlaziywa kwamandla wendabuko zinikezwa Cohen (1988) (incwadi) Cohen (1992) (isihloko), kanti Gelman and Carlin (2014) banikeza umbono ohlukile. I-covariates yangaphambi kokwelapha ingafakwa ekuklanyweni nasekuhlaziyweni kwesigaba sokuhlola; isahluko 4 se- Gerber and Green (2012) sinikeza isingeniso esihle kuzo zonke izindlela, kanti Casella (2008) unikeza ukwelashwa okujulile. Amasu asebenzisa lolu lwazi lwe-pre-treatment ngendlela yokuhlelwa kwemvelo ngokuvamile abizwe ngokuthi i-designs yokuvivinya evinjiwe noma imiklamo yokuhlola ehlelekile (lesi sigama asisetshenziswanga njalo emiphakathini yonke); lezi zindlela zihlobene eduze namasu Higgins, Sävje, and Sekhon (2016) esahlukweni 3. Bheka Higgins, Sävje, and Sekhon (2016) ukuze uthole okuningi ngokusebenzisa lezi zakhiwo ekuhlolweni okukhulu. I-covariate yangaphambi kokwelashwa ingabuye ifakwe kwisigaba sokuhlaziywa. McKenzie (2012) uhlola indlela eyahlukana-ngokungafani ekuhlaziyeni ukuhlolwa kwensimu ngokuningiliziwe. Bheka Carneiro, Lee, and Wilhelm (2016) ukuze uthole okuningi mayelana nokuhwebelana phakathi kwezinyathelo ezahlukene zokwandisa ukucaca ekulinganisweni kwemiphumela yokwelashwa. Okokugcina, lapho unquma ukuthi uzama ukufaka ama-covariates wangaphambi kokwelashwa ekuklanyeni noma ekuhlaziyweni kwesigaba (noma kokubili), kunezici ezimbalwa okufanele uzicabangele. Esikhathini lapho abacwaningi befuna khona ukukhombisa ukuthi "abadobi" (Humphreys, Sierra, and Windt 2013) , ukusebenzisa ama-covariate wangaphambi kokwelashwa esigabeni sokuklama kungasiza (Higgins, Sävje, and Sekhon 2016) . Ezimweni lapho abahlanganyeli befika khona ngokulandelana, ikakhulukazi ukuhlolwa kwenkambu e-intanethi, ukusebenzisa ulwazi lwe-pre-treatment esiteji sokuklama kungase kube nzima kakhulu; bheka, isibonelo, i- Xie and Aurisset (2016) .

Kufanelekile ukufaka intuition encane yokuthi kungani ukuhlukahluka kokungafani-ndlela kungasebenza kakhulu kunomthelela-okushiwo oyedwa. Imiphumela eminingi ye-intanethi inehlukahluka kakhulu (bheka isib. RA Lewis and Rao (2015) Lamb et al. (2015) ) futhi zizinzile ngokuhamba kwesikhathi. Kulokhu, ukuguqulwa kwesilinganiso kuzoba nokuhluka okuncane kakhulu, okwandisa amandla we-test statistics. Esinye sezizathu ukuthi le ndlela ayisebenzisi kaningi yilokho ngaphambi kweminyaka yobudala, kwakungavamile ukuba nemiphumela yokwelapha. Indlela ekhonakala kakhulu yokucabanga ngalokhu ukucabanga ukuzama ukukala ukuthi isimiso esithile sokuzivocavoca sibangela ukulahlekelwa isisindo. Uma uthola indlela ehlukile-in-means, ukulinganisela kwakho kuzoba nokushintshashintsha okuvela ekuhlukeni kwezilinganiso kubantu. Uma wenza umehluko-umahluko, noma kunjalo, lokho okwenzeka ngokwezifiso ezilinganisweni kususiwe, futhi ungakwazi ukuthola kalula umehluko obangelwa ukwelashwa.

Ekugcineni, ngicabanga ukuthi nginezela i-R yesine: "ukuphindaphinda". Okungukuthi, uma abacwaningi bethola ulwazi oluningi lokuhlola kunokuba badinga ukubhekana nombuzo wabo wokucwaninga wangempela, kufanele babuyekeze idatha ukuze babuze imibuzo emisha. Isibonelo, cabanga ukuthi i-Kramer kanye nozakwethu basebenzise umlinganisi wokuhlukana okungafaniyo futhi bathole ngokwedatha eyengeziwe kunalokho okudingekayo ukubhekana nombuzo wabo wocwaningo. Esikhundleni sokungasebenzisi idatha ngokugcwele, bebengase bafunde ubukhulu bomphumela njengendlela yokukhuluma kwangaphambi kokwelapha ngokomzwelo. Njengoba nje Schultz et al. (2007) wathola ukuthi umphumela wokwelashwa wawuhlukile kubasebenzisi abanobucayi nabakhulu, mhlawumbe imiphumela ye-News Feed yayingafani kubantu abavele bejwayele ukuthumela imilayezo ejabulisayo (noma ebuhlungu). Ukuphindaphinda kungabangela "ukudoba" (Humphreys, Sierra, and Windt 2013) "p-hacking" (Simmons, Nelson, and Simonsohn 2011) , kodwa lokhu kukhulunywa kakhulu ngokuhlanganisa ukuthembeka (Simmons, Nelson, and Simonsohn 2011) , ngaphambi kokubhalisa (Humphreys, Sierra, and Windt 2013) , nezindlela zokufunda zomshini ezizama ukugwema ukuphelelwa isikhathi.