Seo u lokelang ho se bala ka mor'a moo
- Selelekela (karolo ea 4.1)
Lipotso tse mabapi le ho etsa lipatlisiso liphuputsong tsa sechaba hangata li rarahane ebile li thata. Bakeng sa mokhoa o ts'oanang oa ho etsa lintho ka tsela e thehiloeng ho lipapali tse bonts'itsoeng, bona Pearl (2009) , le bakeng sa mokhoa o thehiloeng ho latela liphello tse ka bang teng, bona Imbens and Rubin (2015) . Bakeng sa papiso pakeng tsa mekhoa ena e 'meli, sheba Morgan and Winship (2014) . Bakeng sa mokhoa o tloaelehileng oa ho hlalosa ho ferekanya, bona VanderWeele and Shpitser (2013) .
Khaolong ena, ke bōpile se neng se bonahala eka ke moeli o khanyang pakeng tsa bokhoni ba rona ba ho etsa litekanyetso tsa ho hlahisa lintlha tse tsoang litsing tsa liteko le tseo e seng tsa teko. Leha ho le joalo, ke nahana hore, ha e le hantle, phapang e fapane haholo. Ka mohlala, e mong le e mong oa lumela hore ho tsuba ho baka kankere, le hoja ho se na liteko tse laoloang ka tsela e sa sebetseng tse etsang hore batho ba tsube ho se ho fetile. Bakeng sa mekhoa e metle ea phekolo ea nako ea ho bala ka ho etsa likhakanyo tse fokolang tsa lits'ebeletso tse seng tsa teko, bona Rosenbaum (2002) , ( ??? ) , Shadish, Cook, and Campbell (2001) , le Dunning (2012) .
Khaolo ea 1 le ea 2 ea Freedman, Pisani, and Purves (2007) fana ka kenyelletso e hlakileng ea phapang lipakeng tsa liteko, liteko tse laoloang, le liteko tse laoloang ke randomised.
Manzi (2012) fana ka kenyelletso e tsotehang le e utloahalang litabeng tsa filosofi le lipalo-palo tsa liteko tse laoloang ka hohle-hohle. E boetse e fana ka mehlala e tsotehang ea lefatše ea matla a ho leka lits'ebetsong. Issenberg (2012) fana ka kenyelletso e hlollang ea tšebeliso ea liteko litabeng tsa lipolotiki.
- Liteko ke life? (karolo 4.2)
Box, Hunter, and Hunter (2005) , @ casella_statistical_2008, le Athey and Imbens (2016b) fana ka litlhaloso tse ntle litabeng tsa litekanyetso tsa tlhahlobo le tlhahlobo ea liteko. Ho feta moo, ho na le mekhoa e metle ea ho sebelisa liteko litsing tse fapaneng tse fapaneng: moruo (Bardsley et al. 2009) , sociology (Willer and Walker 2007; Jackson and Cox 2013) , psychology (Aronson et al. 1989) , saense ea lipolotiki (Morton and Williams 2010) , le leano la sechaba (Glennerster and Takavarasha 2013) .
Bohlokoa ba ho hira litho tsa sehlopha (mohlala, mohlala) hangata ha bo anane le patlisiso ea liteko. Leha ho le joalo, haeba phello ea phekolo e fapane ho batho, joale sampling e bohlokoa. Longford (1999) e bolela ntlha ena ka mokhoa o hlakileng ha a buella bafuputsi ba nahanne ka liteko ha e le phuputso ea baahi e nang le litekanyetso tse sa tšoaneng.
- Litekanyo tse peli tsa liteko: lebala-le-analog-digital (karolo 4.3)
Ke fane ka tlhahiso ea hore ho na le ts'ebetso e pakeng tsa liteko le liteko tsa tšimo, 'me bafuputsi ba bang ba hlahisitse litlhaloso tse qaqileng, haholo-holo tse arohanyang mefuta e sa tšoaneng ea liteko tsa tšimo (Harrison and List 2004; Charness, Gneezy, and Kuhn 2013) .
Lipampiri tse 'maloa li bapisitse lab le liteko tsa tšimo libakeng tse hlakileng (Falk and Heckman 2009; Cialdini 2009) le ka lipatlisiso tsa liteko tse itseng tsa saense tsa lipolotiki (Coppock and Green 2015) , moruo (Levitt and List 2007a, 2007b; Camerer 2011; Al-Ubaydli and List 2013) , le psychology (Mitchell 2012) . Jerit, Barabas, and Clifford (2013) fana ka lipatlisiso tse ntle tsa ho etsa lipatlisiso ho bapisa liphello tsa liteko le liteko tsa tšimo. Parigi, Santana, and Cook (2017) hlalosa kamoo liteko tsa maqephe a marang-rang li ka kopanyang tse ling tsa litšoaneleho tsa laboraka le tšimo.
Ho tšoenyeha ka barupeluoa ho fetola boitšoaro ba bona hobane ba tseba hore ba ntse ba bolokoa ka hloko ka nako e 'ngoe ba bitsoa liphello tsa tlhokahalo ,' me ba ithutile ka kelelloji (Orne 1962) le moruo (Zizzo 2010) . Le hoja boholo ba tsona bo amahanngoa le liteko tsa laboratoire, lintlha tsena li ka baka mathata le liteko tsa tšimo. Ha e le hantle, liphello tse ling li boetse li bitsoa liphello tsa Hawthorne , e leng lentsoe le fumanang liteko tse tummeng tsa ho khantša tse qalileng ka 1924 ho Hawthorne Works ea Western Electric Company (Adair 1984; Levitt and List 2011) . Bobeli bo hloka liphello le liphello tsa Hawthorne li amana haufi-ufi le khopolo ea mekhoa ea boithaopo e tšohliloeng khaolong ea 2 (bona le Webb et al. (1966) ).
Liteko tsa tšimo li na le histori e telele ea moruo (Levitt and List 2009) , saense ea lipolotiki (Green and Gerber 2003; Druckman et al. 2006; Druckman and Lupia 2012) , psychology (Shadish 2002) le leano la sechaba (Shadish and Cook 2009) . Sebaka se seng sa saense sa sechaba moo liteko tsa tšimo li ileng tsa qala ho hlahella ka potlako ke nts'etsopele ea machaba. Bakeng sa tlhahlobo e ntle ea mosebetsi oo ka hare ho moruo bona Banerjee and Duflo (2009) , le bakeng sa tlhahlobo e boima, bona Deaton (2010) . Bakeng sa mosebetsi ona oa lipolotiki, bona Humphreys and Weinstein (2009) . Qetellong, melao ea boitšoaro e hlahisitsoeng ke liteko tsa tšimo e 'nile ea hlahlojoa maemong a thuto ea lipolotiki (Humphreys 2015; Desposato 2016b) le moruo oa ntlafatso (Baele 2013) .
Karolong ena, ke ile ka fana ka tlhahiso ea hore tlhahisoleseding ea pele ho phekolo e ka sebelisoa ho ntlafatsa ho nepahala ha litekanyetso tsa phekolo, empa ho na le phehisano e mabapi le mokhoa ona; sheba Freedman (2008) , W. Lin (2013) , Berk et al. (2013) , le Bloniarz et al. (2016) bakeng sa tlhahisoleseding e eketsehileng.
Qetellong, ho na le mefuta e meng e 'meli ea liteko tse entsoeng ke bo-rasaense ba sechaba tse sa lumellaneng ka mokhoa o nepahetseng lehlakoreng la li-lab: liteko tsa lipatlisiso le liteko tsa sechabeng. Liteko tsa lipatlisiso ke liteko tse sebelisang mekhoa ea mekhoa ea lipatlisiso le ho bapisa likarabo le liphetoho tse ling tsa lipotso tse tšoanang (liteko tse ling tsa lipatlisiso li hlahisoa Khaolong ea 3); bakeng sa lintlha tse eketsehileng ka liteko tse entsoeng ka lipatlisiso bona Mutz (2011) . Liteko tsa sechaba ke liteko tseo kalafo e leng leano le leng la sechaba le ka sebelisoang ke 'muso feela. Liteko tsa sechaba li amana haholo le tlhahlobo ea lenaneo. Bakeng sa lintlha tse eketsehileng mabapi le liteko tsa pholisi, bona Heckman and Smith (1995) , Orr (1998) , le @ glennerster_running_2013.
- Ho tsamaea ka liteko tse seng kae feela (karolo 4.4)
Ke khethile ho tsepamisa mohopolo lintlheng tse tharo: ts'ebetso, ho haneloa ha liphello tsa phekolo, le lits'ebetso. Mantsoe ana a na le mabitso a sa tšoaneng maemong a fapaneng. Ka mohlala, litsebi tsa kelello li atisa ho fetela lintlheng tse seng kae feela ka ho lebisa tlhokomelo ho babuelli le batlatsi (Baron and Kenny 1986) . Khopolo ea baphatlalatsi e hapuoe ke seo ke se bitsang mekhoa, mme khopolo ea batlatsili e hapuoe ke seo ke se bitsang ho nepahala ha kantle (mohlala, sephetho sa teko e tla ba se fapaneng haeba se ne se matha maemong a fapaneng) le ho haneloa ha liphello tsa phekolo ( mohlala, liphello li kholoanyane bakeng sa batho ba bang ho feta ba bang).
Teko ea Schultz et al. (2007) bontša hore na likhopolo-taba tsa sechaba li ka sebelisoa joang ho theha mehato e sebetsang. Bakeng sa khang e kholo ka karolo ea khopolo ea ho theha mehato e atlehang, sheba Walton (2014) .
- Tlhaloso (karolo 4.4.1)
Likhopolo tsa ho netefatsa ka hare le ka ntle li ile tsa qalisoa ke Campbell (1957) . Sheba Shadish, Cook, and Campbell (2001) bakeng sa histori e qaqileng haholoanyane le ho ela hloko ka hloko lipalo-palo tse nepahetseng, ho netefatsa ka hare, ho haha ho nepahala, le ho netefatsa ka ntle.
Bakeng sa tlhahlobo-palo ea litaba tse amanang le lipapali tsa lipalo-palo li nepahetseng lits'ebetsong, bona Gerber and Green (2012) (ho tloha ka pono ea sechaba sa saense) le Imbens and Rubin (2015) (ho tloha pontsong ea lipalo). Lintlha tse ling tsa ho phethahala ha lipalo-palo tse hlahang ka ho khetheha litsing tsa lits'ebetso tsa marang-rang li kenyeletsa litaba tse kang mekhoa e sebetsang ea ho iketsetsa tšepo ka nako e nepahetseng (Bakshy and Eckles 2013) .
Tšebeliso ea ka hare e ka ba boima ho netefatsa lits'ebetsong tse rarahaneng tsa tšimo. Bona, mohlala, Gerber and Green (2000) , Imai (2005) , le Gerber and Green (2005) bakeng sa moqoqo ka ts'ebetsong ea teko e rarahaneng ea tšimo mabapi le ho vouta. Kohavi et al. (2012) le Kohavi et al. (2013) fana ka kenyelletso mathateng a ho netefatsa nako nakong ea liteko tsa maqephe a marang-rang.
Ntho e 'ngoe e kotsing ea ho netefatsa ka hare ke monyetla oa ho hlōleha ho etsa mesebetsi e mengata. Tsela e 'ngoe e ka fumanoang ea ho lemoha mathata ka mokhoa oa ho etsa lintho ka tloaelo ke ho bapisa lihlopha tsa kalafo le tsa ho laola litšoanelehong tse bonahalang. Mofuta ona oa papiso o bitsoa ho leka-lekaneng . Sheba Hansen and Bowers (2008) bakeng sa mokhoa oa ho hlahloba Mutz and Pemantle (2015) le Mutz and Pemantle (2015) bakeng sa ho ameha ka litekanyetso tsa litekanyetso. Ka mohlala, ho hlahloba ho leka-lekaneng, Allcott (2011) fumane bopaki bo bong ba hore ho fetoha ha mokhoa o mong le o mong ha hoa kenngoa ka mokhoa o nepahetseng ka liteko tse tharo tsa Opower (bona tafole 2; libaka 2, 6 le 8). Bakeng sa litsela tse ling, sheba khaolo ea 21 ea Imbens and Rubin (2015) .
Mathata a mang a maholo a amanang le ho netefala ka hare ke ona: (1) ho se lumellane ka lehlakoreng le le leng, moo batho bohle ba sehlopha sa phekolo ba sa amoheleng kalafo, (2) ho se lumellane ho peli, moo bohle ba leng sehlopha sa phekolo ba amohelang kalafo le batho ba bang sehlopha sa taolo se fumana phekolo, (3) mokhoa oa ho ba le tšebetso, moo liphello li sa lekanyetsoang bakeng sa barupeluoa ba bang, le (4) ho kena-kenana, moo phekolo e tšeloang ho tsoa ho batho ba boemo ba ho phekola ho batho ba boemong ba ho laola. Bona khaolo ea 5, 6, 7 le 8 ea Gerber and Green (2012) bakeng sa ho hongata ho tse ling tsa tsena.
Bakeng sa lintlha tse eketsehileng tabeng ea ho haha ho netefatsa, sheba Westen and Rosenthal (2003) , le bakeng sa ho haha ho netefatsa mehloli e mengata ea litaba, Lazer (2015) le khaolo ea 2 ea buka ena.
Karolo e 'ngoe ea ho netefatsa kantle ke boemo boo ho kenelletsoang ho kenngoa ho lona bo lekoa. Allcott (2015) fana ka tlhokomelo e hlokolosi ea khopolo le e bontšang boikutlo ba khethollo ea sebaka sa khetho. Makasine ena e boetse e tšohloa ke Deaton (2010) . Ntho e 'ngoe ea ho netefatsa ka ntle ke hore na mekhoa e meng e sebelisoang ea ho kenella e tšoanang e tla ba le liphello tse tšoanang. Tabeng ena, papiso pakeng tsa Schultz et al. (2007) le Allcott (2011) bontša hore liteko tsa Opower li na le phello e fokolang ea phekolo ho feta liteko tsa pele tsa Schultz le basebetsi-'moho (1,7% ho ea ho 5%). Allcott (2011) maikutlo a hore liteko tsa ho latela li bile le phello e nyenyane ka lebaka la litsela tseo phekolo e fapaneng ka tsona: emoticon e ngotsoeng ka letsoho e le karolo ea thuto e tšehetsoeng ke univesithi, ha e bapisoa le emoticon e hatisitsoeng e le karolo ea boitsebiso bo bongata tlaleho e tsoang k'hamphaning ea matla.
- Heterogeneity ea liphello tsa phekolo (karolo 4.4.2)
Bakeng sa tlhahlobisiso e hlollang ea phetoho ea phekolo ea phekolo ea tšoaetso tšimong, sheba khaolo ea 12 ea Gerber and Green (2012) . Bakeng sa likenyelletso tsa ho fokotsa ha liphello tsa kalafo litsing tsa bongaka, bona Kent and Hayward (2007) , Longford (1999) , le Kravitz, Duan, and Braslow (2004) . Tlhokomelo ea liphetoho tsa phekolo ea phekolo e atisa ho lebisa tlhokomelo ho liphapang tse itšetlehileng ka litšoaneleho tsa pele ho phekolo. Haeba u thahasella ho fokotsa haterogeneity e thehiloeng liphellong tsa morao-rao, joale ho hlokahala mekhoa e rarahaneng haholoanyane, e kang ho (Frangakis and Rubin 2002) lihlooho tse khōlō (Frangakis and Rubin 2002) ; bona Page et al. (2015) bakeng sa tlhahlobo.
Bafuputsi ba bangata ba hakanya hore ho na le liphello tse fapaneng tsa liphello tsa phekolo tse sebelisoang ka tsela e tloaelehileng, empa mekhoa e mecha e itšetlehile ka ho ithuta ka mochine; bona, mohlala, Green and Kern (2012) , Imai and Ratkovic (2013) , Taddy et al. (2016) , le Athey and Imbens (2016a) .
Ho na le lipelaelo tse mabapi le lipatlisiso tsa mathata a fapaneng le ho "tšoasa litlhapi." Ho na le mekhoa e fapaneng ea lipalo tse ka thusang ho rarolla mathata ka lipapiso tse ngata (Fink, McConnell, and Vollmer 2014; List, Shaikh, and Xu 2016) . Tsela e 'ngoe ea ho tšoenyeha mabapi le "ho tšoasa litlhapi" ke ho ngolisa pele, e leng ho tloaelehileng haholoanyane ho ea ka kelello (Nosek and Lakens 2014) , saense ea lipolotiki (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , le moruo (Olken 2015) .
Phuputsong ea Costa and Kahn (2013) hoo e ka bang halofo feela ea malapa a tekong e ka kopanngoa le tlhahisoleseding ea batho. Babali ba thahasellang lintlha tsena ba lokela ho bua ka pampiri ea pele.
- Mekhoa (karolo 4.4.3)
Mekhoa e bohlokoa haholo, empa e ba thata haholo ho ithuta. Lipatlisiso mabapi le mekhoa e amana haholo le ho ithuta lipakeng tsa li-psychology (empa bona hape VanderWeele (2009) bakeng sa papiso e nepahetseng lipakeng tsa maikutlo a mabeli). Mekhoa ea ho fumana litsela, e kang mokhoa o hlahisitsoeng Baron and Kenny (1986) , o tloaelehile haholo. Ka bomalimabe, ho bonahala hore mekhoa eo e itšetlehile ka likhopolo tse matla (Bullock, Green, and Ha 2010) mme ba utloa bohloko ha ho na le mekhoa e mengata, joalo ka motho a ka lebella maemong a mangata (Imai and Yamamoto 2013; VanderWeele and Vansteelandt 2014) . Imai et al. (2011) le Imai and Yamamoto (2013) fana ka mekhoa e meng e ntlafetseng ea lipalo-palo. Ho feta moo, VanderWeele (2015) fana ka kalafo ea bolelele ba buka ka liphello tse ngata tsa bohlokoa, ho kenyelletsoa mokhoa o felletseng oa tlhahlobo ea kutloisiso.
Tsela e fapaneng e lebisa tlhokomelo lits'ebetsong tse lekang ho sebelisa mochine ka kotloloho (mohlala, ho fa basesisi ba likepe vithamine C). Ka bomalimabe, lits'ebetsong tse ngata tsa saense, hangata ho na le mekhoa e mengata 'me ho thata ho theha mekhoa ea phekolo e fetohang e sa fetole tse ling. Tse ling tse atamela mekhoa ea ho fetola liteko li hlalosoa ke Imai, Tingley, and Yamamoto (2013) , Ludwig, Kling, and Mullainathan (2011) , le Pirlott and MacKinnon (2016) .
Bafuputsi ba sebetsang litekong tse feletseng tsa tlhaho ba tla hloka ho ameha ka tlhahlobo ea maikutlo a mangata; sheba Fink, McConnell, and Vollmer (2014) le List, Shaikh, and Xu (2016) bakeng sa tlhahisoleseling e eketsehileng.
Qetellong, mekhoa e boetse e na le histori e telele filosofi ea saense e hlalositsoeng ke Hedström and Ylikoski (2010) .
- Ho sebelisa maemo a teng (karolo 4.5.1)
Bakeng sa lintlha tse ngata mabapi le tšebeliso ea lithuto tsa mangolo le lithuto tsa ho hlahloba ho hlahloba khethollo, sheba Pager (2007) .
- Iketsetse liteko (karolo ea 4.5.2)
Tsela e tloaelehileng ka ho fetisisa ea ho kopa barupeluoa litekong tseo u li etsang ke Amazon Mechanical Turk (MTurk). Hobane MTurk e etsisa likarolo tsa batho ba tloaelehileng ba lekang ho leka ho etsa mesebetsi eo ba ke keng ba e etsa bakeng sa mahala-bafuputsi ba bangata ba se ba qalile ho sebelisa Turkers (basebetsi ba MTurk) e le barupeluoa ba nang le liteko, e leng se hlahisang pokello ea lits'ebeletso ka potlako le e theko e tlaase ho feta e ka finyelloang litloaelo tsa laboratori (Paolacci, Chandler, and Ipeirotis 2010; Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012; Rand 2012; Berinsky, Huber, and Lenz 2012) .
Ka kakaretso, menyetla e meholo ka ho fetisisa ea ho sebelisa barupeluoa ba fumanoang ho tloha MTURK ke lisebelisoa. Le hoja liteko tsa li-laboratory li ka nka libeke hore li sebetse le liteko tsa tšimo li ka nka likhoeli ho itokisetsa, liteko tsa ba kenang ho tloha MTURK li ka tsamaisoa ka matsatsi. Ka mohlala, Berinsky, Huber, and Lenz (2012) ba ile ba khona ho bokella lithuto tse 400 ka letsatsi le le leng hore ba kopanele liteko tsa metsotso e 8. Ho feta moo, barupeluoa bana ba ka ngolisoa bakeng sa morero leha e le ofe (ho kenyelletsa lipatlisiso le tšebelisano-'moho le batho ba bangata, joalokaha ho boletsoe khaolong ea 3 le ea 5). Ts'ebetso ena ea boiketlo e bolela hore bafuputsi ba ka tsamaisa litekanyetso tse amanang le liteko tse amanang le tsona ka potlako.
Pele u ngolisa barupeluoa ba tsoang MTURK bakeng sa liteko tsa hau, ho na le lintho tse 'nè tsa bohlokoa tseo u lokelang ho li tseba. Ntlha ea pele, bafuputsi ba bangata ba na le lipelaelo tse sa reng letho tsa liteko tse amanang le Turkers. Hobane ts'oaetso ena ha e totobetse, ho thata ho loantša bopaki. Leha ho le joalo, ka mor'a lilemo tse 'maloa tsa lithuto tsa ho sebelisa Turkers, re ka fihlela qeto ea hore ho belaella ho joalo ha ho na lebaka le utloahalang. Ho 'nile ha e-ba le liphuputso tse ngata ho bapisa palo ea baahi ba Turkers le ba batho ba bang le lipatlisiso tse ngata ho bapisa liphello tsa liteko tseo ba Turkers ba li fumanang ho batho ba bang. Ha ke filoe mosebetsi oohle, ke nahana hore tsela e ntle ka ho fetisisa eo u ka nahanang ka eona ke hore Turkers ke sampula e loketseng, e kang liithuti empa tse fapaneng haholo (Berinsky, Huber, and Lenz 2012) . Kahoo, feela joalokaha liithuti e le baahi ba nang le kahlolo e nepahetseng ho ba bang, empa hase bohle, lipatlisiso, Turkers ke baahi ba nang le kahlolo e itseng ho ba bang, empa hase bohle, lipatlisiso. Haeba u il'o sebetsa le Turkers, joale hoa utloahala ho bala tse ngata tsa lithuto tsena ho bapisa le ho utloisisa mekhoa ea bona.
Ntlha ea bobeli, bafuputsi ba iketsetse mekhoa e metle bakeng sa ho eketsa liteko tsa MTurk ka hare, 'me u lokela ho ithuta le ho latela litloaelo tsena tse ntle (Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012) . Ka mohlala, bafuputsi ba sebelisang Turkers ba khothalletsoa hore ba sebelise li-screener ho tlosa barupeluoa ba sa (Berinsky, Margolis, and Sances 2014, 2016) (empa bona DJ Hauser and Schwarz (2015b) le DJ Hauser and Schwarz (2015a) ). Haeba u sa tlose barupeluoa ba sa mameleng, joale phello leha e le efe ea phekolo e ka hlatsoa ka lerata leo ba e kenyelletsang, 'me ka ho etsa joalo palo ea barupeluoa ba sa mameleng e ka ba ea bohlokoa. Ha Huber le basebetsi-'moho le eena ba lekoa (2012) , hoo e ka bang karolo ea 30 lekholong ea barupeluoa e ile ea hlōleha ho ela hloko lintlha tsa motheo. Mathata a mangata ao hangata a hlahang ha basebetsi ba Turtle ba sebelisoa ke barupeluoa ba se nang bolulo (Chandler et al. 2015) le attrition (Zhou and Fishbach 2016) .
Ea boraro, ho latela mefuta e meng ea liteko tsa digital, liteko tsa MTurk li ke ke tsa hōla; Stewart et al. (2015) lekanyetsa hore nako efe kapa efe ho na le batho ba ka bang 7 000 feela ba MTurk.
Qetellong, o lokela ho tseba hore MTurk ke setsi se nang le melao le melao ea sona (Mason and Suri 2012) . Ka tsela e tšoanang eo u ka leka ho e tseba ka setso sa naha eo u neng u il'o leka liteko tsa hau, u lokela ho leka ho fumana ho eketsehileng ka setso le mekhoa ea Turkers (Salehi et al. 2015) . 'Me u lokela ho tseba hore ba Turkers ba tla be ba bua ka teko ea hau haeba u etsa ntho e sa lokelang kapa e sa nepahaleng (Gray et al. 2016) .
MTurk ke mokhoa o tsotehang oa ho kopa barupeluoa lits'ebetsong tsa hau, ebang ke li-Lab Huber, Hill, and Lenz (2012) , kapa tse ling tse kang tsa Mason and Watts (2009) , Goldstein, McAfee, and Suri (2013) , Goldstein et al. (2014) , Horton and Zeckhauser (2016) , le Mao et al. (2016) .
- Iketsetse sehlahisoa sa hau (karolo 4.5.3)
Haeba u nahana ho leka ho iketsetsa sehlahisoa sa hau, ke khothaletsa hore u bale keletso e fanoang ke sehlopha sa MovieLens Harper and Konstan (2015) . Tlhaloso ea bohlokoa ho tsoa phihlelong ea bona ke hore bakeng sa morero o mong le o mong o atlehileng ho na le mefokolo e mengata. Ka mohlala, sehlopha sa MovieLens se hlahisitse lihlahisoa tse ling, tse kang GopherAnswers, tse neng li hlōleha ka ho feletseng (Harper and Konstan 2015) . Mohlala o mong oa mofuputsi oa hlōleha ha a leka ho haha sehlahisoa ke boiteko ba Edward Castronova ea ho haha papali ea Inthanete e bitsoang Arden. Ho sa tsotellehe lichelete tsa lichelete tsa $ 250,000, morero ona o ne o phaphametse (Baker 2008) . Merero e kang GopherAnswers le Arden ka bomalimabe e atile haholo ho feta merero e kang MovieLens.
- Ho sebelisana le ba matla (karolo 4.5.4)
Ke utloile mohopolo oa Quadrant ea Pasteur e tšohloang hangata lik'hamphani tsa theknoloji, mme o thusa ho hlophisa boiteko ba lipatlisiso ho Google (Spector, Norvig, and Petrov 2012) .
Bond le basebetsi-'moho 'lithuto (2012) bona ba leka ho lemoha phello ea mekhoa ena ea phekolo ho metsoalle ea ba e amohetseng. Ka lebaka la tlhahlobo ea teko, li-spillovers tsena li thata ho bona hore li hloekile; Babali ba thahasellang ba lokela ho bona Bond et al. (2012) bakeng sa puisano e tebileng. Jones le basebetsi-'moho (2017) bona ba ile ba etsa liteko tse tšoanang nakong ea likhetho tsa 2012. Liteko tsena ke karolo ea litekanyetso tse telele tsa liteko tsa saense tsa lipolotiki ka boiteko ba ho khothaletsa ho vouta (Green and Gerber 2015) . Liteko tsena li fumaneha, ka kakaretso hobane li le Pasteur ea Quadrant. Ke hore, ho na le batho ba bangata ba susumetsoang ho eketsa likhetho le ho vouta e ka ba boitšoaro bo thahasellisang ho leka likhopolo tse eketsehileng ka phetoho ea boitšoaro le tšusumetso ea sechaba.
Bakeng sa keletso mabapi le ho etsa litekanyetso tsa tšimo le mekhatlo e kopanetsoeng e kang mekhatlo ea lipolotiki, mekhatlo e ikemetseng, le likhoebo, bona Loewen, Rubenson, and Wantchekon (2010) , JA List (2011) , le Gueron (2002) . Bakeng sa mehopolo mabapi le hore na likamano le mekhatlo li ka ama mekhoa ea lipatlisiso joang, bona King et al. (2007) le Green, Calfano, and Aronow (2014) . Boikarabello bo ka boela ba lebisa lipotsong tsa boitšoaro, joalokaha ho builoe ke Humphreys (2015) le Nickerson and Hyde (2016) .
- Keletso ea tlhophiso (karolo 4.6)
Haeba u etsa moralo oa ts'ebetso pele o sebetsana le liteko tsa hau, ke fana ka maikutlo a hore u qale ka ho bala melao-motheo ea litlaleho. Melao-motheo ea CONSORT (Consolidated Standard Reporting of Trials) e ile ea hlahisoa meriana (Schulz et al. 2010) mme e fetotsoe lipatlisisong tsa sechaba (Mayo-Wilson et al. 2013) . Sehlopha se latelang sa tataiso se entsoe ke bahlophisi ba Journal of Experimental Political Science (Gerber et al. 2014) (bona le Mutz and Pemantle (2015) le Gerber et al. (2015) ). Qetellong, litlaleho tsa litlaleho li entsoe ka psychology (APA Working Group 2008) , 'me u bone le Simmons, Nelson, and Simonsohn (2011) .
Haeba u etsa moralo oa tlhahlobo, o lokela ho nahana ka pele ho e ngolisa hobane pele ho ngolisoang ho tla eketsa tšepo eo ba bang ba nang le eona liphellong tsa hau. Ho feta moo, haeba u sebetsa le molekane, e tla fokotsa bokhoni ba molekane oa hao ho fetola tlhahlobo ka mor'a ho bona liphetho. Pele ho ngolisoang e ntse e atisa ho eketseha litabeng tsa kelello (Nosek and Lakens 2014) , saense ea lipolotiki (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , le moruo (Olken 2015) .
Keletso e entsoeng ka ho khetheha bakeng sa liteko tsa Konstan and Chen (2007) marang-rang e boetse e hlahisoa Konstan and Chen (2007) le Chen and Konstan (2015) .
Seo ke se bitsitseng leano la armada ka linako tse ling li bitsoa lipatlisiso tsa lenaneo ; sheba Wilson, Aronson, and Carlsmith (2010) .
- Etsa data e fapaneng ea litšenyehelo (karolo 4.6.1)
Bakeng sa lintlha tse ling tsa liteko tsa MusicLab, bona Salganik, Dodds, and Watts (2006) , Salganik and Watts (2008) , Salganik and Watts (2009b) , Salganik and Watts (2009a) , le Salganik (2007) . Bakeng sa lintlha tse eketsehileng ka limmaraka tse atlehang, bonang Frank and Cook (1996) . Bakeng sa ho eketsehileng ka ho tlosa mahlohonolo le bokhoni ka kakaretso, bona Mauboussin (2012) , Watts (2012) , le Frank (2016) .
Ho na le mokhoa o mong oa ho felisa litefello tsa barupeluoa tseo bafuputsi ba lokelang ho li sebelisa ka hloko: ho ngolisoa. Litsing tse ngata tsa inthaneteng liteko tsa barupeluoa li ngotsoe litekong 'me ha lia ka tsa lefshoa. Mehlala ea mokhoa ona e kenyeletsa tlhahlobo ea Restivo le van de Rijt (2012) bakeng sa meputso ea tlhahlobo ea Wikipedia le Bond le ea (2012) ea ho khothaletsa batho ho vouta. Liteko tsena ha li na litšenyehelo tse fapaneng-ho e-na le hoo, li na le litšenyehelo tse fapaneng tsa bafuputsi . Lits'itsong tse joalo, esita le haeba litšenyehelo ho motho e mong le e mong a le monyenyane haholo, litšenyehelo tse ngata li ka ba kholo haholo. Bafuputsi ba sebetsang litekong tse ngata tsa inthaneteng hangata ba fana ka mabaka a bohlokoa ba mekhoa e menyenyane ea phekolo ka ho re liphello tsena tse nyenyane li ka ba tsa bohlokoa ha li sebelisoa ho batho ba bangata. Ho nahana ka tsela e tšoanang ho sebetsa le litšenyehelong tseo bafuputsi ba li behang ho barupeluoa. Haeba tlhahlobo ea hau e etsa hore batho ba limilione tse ling ba senye metsotso e le 'ngoe, teko ha e kotsi ho motho leha e le ofe, empa ka kakaretso e senyehile hoo e ka bang lilemo tse peli.
Tsela e 'ngoe ea ho bitsa chelete e fapaneng ea litšenyehelo bakeng sa barupeluoa ke ho sebelisa lotho, mokhoa o sebelisitsoeng le lipatlisisong tsa lipatlisiso (Halpern et al. 2011) . Bakeng sa lintlha tse ngata mabapi le ho etsa liphihlelo tse thabisang tsa basebelisi, sheba Toomim et al. (2011) . Bakeng sa lintlha tse eketsehileng mabapi le ho sebelisa bots ho etsa liteko tse sa fetoheng tsa litšenyehelo bona ( ??? ) .
- Etsa, Etsa Hlalosa, 'me U fokotsa (karolo 4.6.2)
Li-R tse tharo tse qalileng li hlahisoa ke Russell and Burch (1959) ke tse latelang:
"Phetolo bolela tloswa bakeng elang hloko phelang liphoofolo phahameng oa lintho tse bonahalang insentient. Phokotso bolela phokotso e palo ea liphoofolo sebelisoa ho fumana tlhahisoleseding ya tjhelete e fuoa le sebetsa ka ho nepahetseng. Ntlafatso ho bolela fokotseha leha e le lefe incidence kapa teba ha mekgwatshebetso ya hlokang botho sebelisoa ho liphoofolo tseo e leng ntse ba lokela ho sebelisoa. "
Li-R tse tharo tseo ke li hlahisang ha li fetele melao ea boitšoaro e hlalositsoeng khaolong ea 6. Ho e-na le hoo, ke phetolelo e hlalositsoeng ka ho fetisisa ea melao-ea bohlokoa-haholo-holo ha ho etsoa liteko tsa batho.
Ho ea ka lekhetho la pele la R ("sebaka"), ho bapisa liteko tsa tšoaetso ea maikutlo (Kramer, Guillory, and Hancock 2014) le tlhahlobo ea maikutlo ea tlhaho (Lorenzo Coviello et al. 2014) fana ka lithuto tse akaretsang mabapi le khoebo e amehang ho tloha litekong ho ea litsing tsa tlhaho (le mekhoa e meng e tšoanang le e lekang ho leka litekanyetso ka lits'ebetso tse seng tsa liteko; bona khaolo ea 2). Ho phaella melemo ea boits'oaro, ho fetoha ho tsoa litekong ho ea liphuputsong tseo e seng tsa liteko ho boetse ho etsa hore bafuputsi ba ithute phekolo eo ba sa khoneng ho e sebelisa. Melemo ena ea boitšoaro le mekhoa e nepahetseng e tla ka litšenyehelo, leha ho le joalo. Ka liteko tsa tlhaho bafuputsi ba na le taolo e fokolang holim'a lintho tse kang ho hira barupeluoa, ho fetoha ha nako, le mofuta oa phekolo. Ka mohlala, moeli o le mong oa pula e le phekolo ke hore o eketsa positivity mme o fokotsa ho hloka matla. Leha ho le joalo, thupelong ea liteko, Kramer le basebetsi-'moho le eena ba ile ba khona ho fetola positivity le ho se tsotelle ka boithati. Tsela e khethehileng e sebelisoang ke Lorenzo Coviello et al. (2014) ile ea ntlafatsoa ka ho eketsehileng ke L. Coviello, Fowler, and Franceschetti (2014) . Bakeng sa kenyelletso ea mefuta-futa ea lihlopha, ke mokhoa o sebelisoang ke Lorenzo Coviello et al. (2014) , bona Angrist and Pischke (2009) (e seng ea molao) kapa Angrist, Imbens, and Rubin (1996) (ho feta molao). Bakeng sa tlhahlobo ea lipelaelo tsa lihlopha, sheba Deaton (2010) , le bakeng sa kenyelletso ea mekhoa ea lihlopha tse nang le lisebelisoa tse fokolang (pula ke sesebelisoa se fokolang), bona Murray (2006) . Ka kakaretso, kenyelletso e ntle ho liteko tsa tlhaho e fanoa ke Dunning (2012) , ha Rosenbaum (2002) , ( ??? ) , le Shadish, Cook, and Campbell (2001) fana ka maikutlo a matle ka ho lekanyetsa liphello tse bakoang ntle le liteko.
Ho ea ka lekhetlo la bobeli R ("ntlafatso"), ho na le saense le lisebelisoa tsa khoebo ha ho buuoa ka ho fetola moqapi oa ho ba le tšoaetso ea maikutlo ho thibela liforomo ho ntlafatsa likarolo. Ka mohlala, ho ka 'na ha e-ba joalo hore tlhahiso-pele ea theknoloji ea News Feed e etsa hore e be bonolo haholoanyane ho etsa teko ea hore na mangolo a thibetsoe ho e-na le a holimo ho eng (e leng tlhahiso ea hore ho leka ho thibela melaetsa ho ka sebelisoa e le sebaka se ka holim'a News Feed tsamaiso ntle le tlhokahalo ea liphetoho tsa tsamaiso ea motheo). Leha ho le joalo, saense, leha ho le joalo, khopolo e boletsoeng ke teko ha ea ka ea bontša ka ho hlaka hore e entsoe ka e 'ngoe ho e' ngoe. Ka bomalimabe, ha ke tsebe ka lipatlisiso tse ngata tsa pele mabapi le melemo ea boleng ba ho thibela le ho matlafatsa litaba ho News Feed. Hape, ha ke e-s'o bone lipatlisiso tse ngata mabapi le ho hloekisa phekolo e le hore li se ke tsa e-ba kotsi; ngoe e ikhethang ke B. Jones and Feamster (2015) , e nahanang ka nyeoe ea tekanyo ea ho hlahloba Inthaneteng (sehlooho seo ke se tšohlang khaolong ea 6 ho amana le thuto ea Encore (Burnett and Feamster 2015; Narayanan and Zevenbergen 2015) ).
Mabapi le karolo ea boraro ea R ("ho fokotseha"), litlhaloso tse ntle tsa tlhahlobo ea matla a setso li fanoa ke Cohen (1988) (buka) le Cohen (1992) (sehlooho), athe Gelman and Carlin (2014) fana ka pono e fapaneng. Li-covariate tsa pele ho phekolo li kenyelelitsoe moqapetsong le tlhahlobo ea liteko; Khaolo ea 4 ea Gerber and Green (2012) fana ka kenyelletso e ntle ho mekhoa eo ka bobeli, 'me Casella (2008) fana ka phekolo e tebileng. Litsebelisoa tse sebelisang tlhahisoleseding ena ea pele ho phekolo ka nako e tloaelehileng li bitsoa liqapi tsa tlhahlobo tse thibelitsoeng kapa tse entsoeng ka lisebelisoa tsa liteko (lebitso lena ha le sebelisoe ka mokhoa o ts'oanelang ho pholletsa le sechaba); mekhoa ena e amana haufi-ufi le mekhoa ea litekanyetso tse khethiloeng ho khaolo ea 3. Bona Higgins, Sävje, and Sekhon (2016) bakeng sa ho sebelisa lisebelisoa tsena ka liteko tse kholo. Li-covariate pele ho kalafo li ka boela tsa kenyelletsoa boemong ba ho hlahloba. McKenzie (2012) hlahloba mokhoa o fapaneng le oa ho fapana ho hlahloba liteko tsa tšimo ka ho qaqileng. Sheba Carneiro, Lee, and Wilhelm (2016) bakeng sa ho hokahana ka mekhoa e fapaneng ea ho eketsa ts'ebetso e nepahetseng ka tekanyo ea liphello tsa phekolo. Qetellong, ha u etsa qeto ea hore na u leka ho kenyeletsa li-covariate pele ho phekolo nakong ea liqapi kapa lipatlisiso (kapa ka bobeli), ho na le lintlha tse 'maloa tseo u lokelang ho nahana ka tsona. Sebakeng seo bafuputsi ba batlang ho bontša hore ha ba "tšoasa litlhapi" (Humphreys, Sierra, and Windt 2013) , ho sebelisa li-covariate pele ho phekolo nakong ea moqapi ho ka ba molemo (Higgins, Sävje, and Sekhon 2016) . Boemong boo ho bona barupeluoa ba fihlang ka sequentially, haholo-holo litlhahlobo tsa masimo a marang-rang, ho sebelisa boitsebiso bo ka pele ho phekolo boemong bo qapiloeng bo ka ba boima haholo; bona, mohlala, Xie and Aurisset (2016) .
Ke habohlokoa ho phaella ka ho hlahisa maikutlo a hore na ke hobane'ng ha mokhoa o fapaneng oa ho fapana ho ka ba o atlehang haholo ho feta phapang-ka-e bolelang. Liphello tse ngata tsa inthaneteng li na le phapang e khōlō (bona mohlala, RA Lewis and Rao (2015) le Lamb et al. (2015) ) 'me li batla li tsitsitse ka mor'a nako. Tabeng ena, palo ea liphetoho e tla ba le phapang e nyane haholo, ho eketsa matla a teko ea lipalo. Lebaka le leng le etsang hore mokhoa ona o sa sebelisoe hangata ke hore pele ho lilemo tsa digital, ho ne ho sa tloaeleha ho ba le liphello tsa pelehi. Tsela e tobileng ea ho nahana ka sena ke ho nahana ka teko ea ho lekanya hore na mokhoa o itseng oa boikoetliso o etsa hore ho be le boima ba 'mele. Haeba o nka mokhoa o fapaneng le oa ho fapana, ho hakanngoa ha hau ho tla ba le phapang e bakoang ke ho fapana ha litekanyo tsa baahi. Haeba u etsa phapang pakeng tsa li-phapang, leha ho le joalo, phapang eo ka tlhaho e etsoang ka litekanyo li tlosoa, 'me u ka lemoha habonolo phapang e bakoang ke phekolo.
Qetellong, ke ile ka nahana ho phaella ka ho reka R: "repurpose". Ka mantsoe a mang, haeba bafuputsi ba iphumana ba e-na le boitsebiso bo eketsehileng ba liteko ho feta kamoo ba lokelang ho rarolla potso ea bona ea lipatlisiso tsa pele, ba lokela ho khutlisetsa data ho botsa lipotso tse ncha. Ka mohlala, nahana hore Kramer le basebetsi-'moho le eena ba ne ba sebelisitse tekanyo ea ho fapana ho fapaneng 'me ba iphumana ba e-na le lintlha tse ngata ho feta kamoo ba neng ba lokela ho rarolla potso ea bona ea lipatlisiso. Ho e-na le hore ba se ke ba sebelisa data ka botlalo, ba ka be ba ithutile boholo ba phello e le mosebetsi oa polelo ea pele ho phekolo ea maikutlo. Feela joalokaha Schultz et al. (2007) fumane hore phello ea phekolo e ne e fapane le basebelisi ba bonolo le ba boima, mohlomong liphello tsa News Feed li ne li fapane le batho ba seng ba ntse ba rata ho ngola melaetsa e thabisang (kapa e utloisang bohloko). Ho pheta-pheta ho ka lebisa ho "ho tšoasa litlhapi" (Humphreys, Sierra, and Windt 2013) le "ho-hacking" (Simmons, Nelson, and Simonsohn 2011) , empa tsena li ka sebetsana haholo le tlaleho e tšepahalang (Simmons, Nelson, and Simonsohn 2011) , pele ho ho ngolisoa (Humphreys, Sierra, and Windt 2013) , le mekhoa ea ho ithuta mechine e lekang ho qoba ho feta.
