Yuav nyeem dab tsi ntxiv
- Kev Taw Qhia (Tshooj 4.1)
Cov nqe lus nug txog kev ua rau hauv kev tshawb nrhiav kev sib haum xeeb feem ntau yog qhov nyuaj thiab sib luag. Rau ib qho kev ua kom pom tseeb los ntawm qhov ua tau zoo, saib Pearl (2009) , thiab rau cov kev ua kom pom tseeb raws li cov txiaj ntsim tau los, saib Imbens and Rubin (2015) . Rau kev sib piv ntawm ob txoj kev no, saib Morgan and Winship (2014) . Rau kev tsim kom muaj cuab kav txhais tau tias muaj kev tawm dag zog, saib VanderWeele and Shpitser (2013) .
Nyob rau hauv tshooj lus no, kuv tau tsim dab tsi zoo li txoj kab sib txawv ntawm peb qhov kev muaj peev xwm los ua kom ntsuas ntawm cov kev sim thiab kev sim uas tsis yog kev sim. Txawm li cas los xij, kuv xav tias, qhov tseeb, qhov txawv yog ntau qhov muag plooj. Piv txwv, txhua tus neeg lees txais tias kev haus luam yeeb ua rau mob cancer, txawm tias tsis muaj kev sim rau kev tswj hwm uas ua rau cov neeg haus luam yeeb tau ua tas lawm. Rau kev kho mob zoo heev ntawm kev ntsuas kev ntsuas ntawm cov ntaub ntawv uas tsis yog kev sim saib Rosenbaum (2002) , ( ??? ) , Shadish, Cook, and Campbell (2001) , thiab Dunning (2012) .
Chapters 1 thiab 2 ntawm Freedman, Pisani, and Purves (2007) qhia meej meej txog cov kev sib txawv ntawm cov kev sim, kev sim kev sim, thiab cov kev sim randomized.
Manzi (2012) muab ib qho kev nthuav dav thiab nyeem tau yooj yim rau cov tswv yim kev tshawb fawb thiab kev tshawb pom ntawm cov kev sim randomized. Nws tseem qhia txog kev nthuav dav ntawm lub ntiaj teb ntawm lub zog ntawm kev lag luam hauv kev lag luam. Issenberg (2012) muab cov lus qhia ntxim nyiam rau kev siv cov kev sim nyob rau hauv nom tswv kev khiav haujlwm.
- Cov kev sim yog dab tsi? (Seem 4.2)
Box, Hunter, and Hunter (2005) , @ casella_statistical_2008, thiab Athey and Imbens (2016b) muab kev qhia zoo rau cov kev sib txuas ntawm kev sim thiab kev tshuaj ntsuam. Ntxiv mus, muaj cov zoo heev kev kho mob ntawm kev siv ntawm thwmsim nyob rau hauv ntau txawv teb: economics (Bardsley et al. 2009) , Sociology (Willer and Walker 2007; Jackson and Cox 2013) , psychology (Aronson et al. 1989) , political science (Morton and Williams 2010) , thiab txoj cai social (Glennerster and Takavarasha 2013) .
Qhov tseem ceeb ntawm tus neeg tuaj koom ua haujlwm (piv txwv, kev ntsuam xyuas) feem ntau muaj kev zoo siab rau kev tshawb nrhiav kev sim. Txawm li cas los xij, yog tias qhov tshwm sim los ntawm kev kho mob yog heterogeneous nyob rau hauv cov pejxeem, ces piv txwv tseem ceeb heev. Longford (1999) ua qhov taw qhia kom meej meej thaum nws tawm tswv yim rau cov kws tshawb fawb xav txog cov kev sim ua ib qho kev tshawb fawb pejxeem nrog kev kuaj zauv.
- Ob qhov kev sim ntawm cov kev sim: lab-field thiab analog-digital (ntu 4.3)
Kuv tau pom hais tias muaj ntau qhov kev sim ntawm cov kev sim (lab) thiab kev tshawb nrhiav, thiab lwm cov neeg tshawb xyuas tau npaj cov ncauj lus kom ntxaws txog ntau yam, tshwj xeeb tshaj yog cov uas cais ntau hom kev sim (Harrison and List 2004; Charness, Gneezy, and Kuhn 2013) .
Muaj ntau cov ntaub ntawv tau muab piv rau cov qauv kev tshawb fawb hauv kev tshawb fawb (Falk and Heckman 2009; Cialdini 2009) thiab ntawm cov txiaj ntsim ntawm cov kev sim hauv kev xaj fai fab (Coppock and Green 2015) , kev lag luam (Levitt and List 2007a, 2007b; Camerer 2011; Al-Ubaydli and List 2013) , thiab psychology (Mitchell 2012) . Jerit, Barabas, and Clifford (2013) muab cov qauv kev tshawb fawb zoo los sib piv cov txiaj ntsim los ntawm cov kev sim (lab) thiab kev ua haujlwm. Parigi, Santana, and Cook (2017) piav qhia txog kev siv thev naus laus zis hauv Internet los ua ke ua ke tau qee yam ntawm cov qauv kev sim thiab cov kev sim.
Kev txhawj xeeb txog cov neeg tuaj yeem hloov lawv tus cwj pwm vim lawv paub hais tias lawv tab tom raug kev pom zoo raug qee zaum hu ua kev tsim txom , thiab lawv tau kawm txog psychology (Orne 1962) thiab kev lag luam (Zizzo 2010) . Txawm hais tias feem ntau yog txuam nrog kev sim (lab), cov teeb meem qub no tuaj yeem tsim teeb meem rau kev sim thwmsim thiab. Qhov tseeb, cov lus nug kuj muaj qee zaum hu ua Hawthorne cov teebmeem , ib lub sij hawm uas muab lub npe hu ua nto npe hu ua 1924 ntawm Hawthorne Works ntawm Western Electric Company (Adair 1984; Levitt and List 2011) . Ob qhov kev thov los cuam tshuam thiab Hawthorne cov teebmeem muaj feem xyuam nrog rau lub tswv yim ntawm kev ntsuas kev sib tham uas tau tham hauv tshooj 2 (saib Webb et al. (1966) ).
Kev tshawb nrhiav teb chaws muaj keeb kwm ntev hauv kev lag luam (Levitt and List 2009) , kev tswjfwm txog kev tswjhwm (Green and Gerber 2003; Druckman et al. 2006; Druckman and Lupia 2012) , psychology (Shadish 2002) , thiab public policy (Shadish and Cook 2009) . Ib cheeb tsam ntawm kev tshawb fawb txog kev ua lag luam uas muaj kev sim nrawm nrawm nrawm yog qhov tseem ceeb thoob ntiaj teb. Rau kev txheeb xyuas qhov kev lag luam hauv kev lag luam no saib Banerjee and Duflo (2009) , thiab kev ntsuam xyuas tseem ceeb pom Deaton (2010) . Rau kev ntsuam xyuas ntawm txoj haujlwm no hauv kev xaj fai fab science saib Humphreys and Weinstein (2009) . Thaum kawg, cov teeb meem tshwm sim los ntawm qhov kev sim ntawm qhov kev tshawb nrhiav tau raug tshawb nrhiav hauv cov ntsiab lus ntawm kev teev hawm nom tswv (Humphreys 2015; Desposato 2016b) thiab kev lag luam tsim (Baele 2013) .
Hauv tshooj lus no, kuv pom tias cov lus qhia ua ntej yuav siv los txhim kho qhov kev txiav txim siab ntawm kev kho mob kwv yees, tab sis muaj qee cov kev sib cav txog qhov no; saib Freedman (2008) , W. Lin (2013) , Berk et al. (2013) , thiab Bloniarz et al. (2016) yog xav paub ntxiv.
Thaum kawg, nws muaj ob hom kev sim uas tau ua los ntawm cov kws tshawb fawb uas tsis haum haum raws li qhov ntsuas hauv tsev (lab): qhov kev tshawb nrhiav thev naus laus zis thiab kev sim ua neej. Cov kev sim cov ntawv soj ntsuam yog sim siv cov khoom siv ntawm cov kev tshawb fawb thiab sim cov lus teb rau lwm cov lus nug ntawm cov lus nug tib yam (ib co kev sim kev tshawb fawb tau nthuav tawm hauv Tshooj 3); rau ntau tshaj rau kev tshawb nrhiav thwmsim pom Mutz (2011) . Kev sim kev sim yog cov kev sim uas qhov kev kho mob yog qee qhov kev sib raug zoo uas tsuas yog siv los ntawm tsoomfwv. Cov kev sim (social) muaj feem xyuam nrog kev ntsuam xyuas kev kawm. Yog xav paub ntxiv txog kev cai sim, saib Heckman and Smith (1995) , Orr (1998) , thiab @ glennerster_running_2013.
- Tsiv tshaj li sim yooj yim (Tshooj 4.4)
Kuv tau xaiv los tsom rau peb lub ntsiab lus: kev siv tau, kev ua haujlwm ntawm kev kho mob, thiab cov tswv yim. Cov ntsiab lus no muaj ntau lub npe hauv ntau hom. Piv txwv li, cov kws kho kev puas siab puas ntsws dhau txoj kev sim yooj yim los ntawm kev teeb tsa kev kho kom haum xeeb thiab cov neeg ua haujlwm (Baron and Kenny 1986) . Lub tswv yim ntawm cov neeg tuaj daws teeb meem yuav raug muab los ntawm qhov kuv hu ua mechanisms, thiab lub tswv yim ntawm cov neeg khiav dej num raug ntes los ntawm qhov kuv hu rau lwm qhov kev siv tau (xws li, cov ntsiab lus ntawm kev sim yuav txawv dua yog tias nws tau khiav hauv ntau zaus) thiab heterogeneity ntawm kev kho mob cov teebmeem ( xws li, yog qhov teebmeem loj dua rau qee cov neeg tshaj rau lwm tus).
Qhov sim los ntawm Schultz et al. (2007) qhia tau hais tias kev siv tus vevxeeb li cas thiaj li siv tau los tsim cov kev pabcuam zoo. Rau kev sib cav ntau ntxiv txog lub luag haujlwm ntawm txoj kev xav ntawm kev tsim cov kev tiv thaiv zoo, saib Walton (2014) .
- Qhov tseeb (seem 4.4.1)
Cov ntsiab lus ntawm sab hauv thiab sab nraud validity tau xub pib los ntawm Campbell (1957) . Saib Shadish, Cook, and Campbell (2001) rau cov keeb kwm muaj tseeb thiab ceev faj txog kev hloov qhov xaus qhov tseeb, sab hauv kev siv, tsim validity, thiab lwm qhov kev siv tau.
Rau cov ntsiab lus muaj feem xyuam rau kev sib nrauj xaus qhov kev siv tau hauv kev sim ua ke saib Gerber and Green (2012) (ntawm kev tshawb fawb txog kev lag luam) thiab Imbens and Rubin (2015) (los ntawm cov qauv pom). Qee qhov teeb meem ntawm kev sib nrauj xaus qhov tseeb uas tshwm sim nyob hauv online thev naus laus zis muaj xws li cov teeb meem xws li kev siv lej lej kev ua kom muaj kev ruaj siab sib txawv nrog cov ntaub ntawv khaws cia (Bakshy and Eckles 2013) .
Internal validity yuav tsis yooj yim kom paub meej hauv txoj kev sim thev plaub. Saib, piv txwv li, Gerber and Green (2000) , Imai (2005) , thiab Gerber and Green (2005) rau kev sib cav txog kev siv txoj kev sim ntawm txoj kev pov npav. Kohavi et al. (2012) thiab Kohavi et al. (2013) muab qhov kev taw qhia rau cov kev cov nyom ntawm kev siv tau ntawm kev ua haujlwm nyob rau hauv kev kawm online.
Ib qho loj kev hem thawj rau hauv kev siv tau yog qhov kev ua tsis tau zoo. Ib txoj hau kev txhawm rau txhawm rau txhawm rau txheeb xyuas qhov teeb meem nrog cov kev xaiv no yog sib piv cov kev kho mob thiab kev tswj hwm ntawm cov kev coj ua. Qhov kev sib piv no yog hu ua tshuav nyiaj tshuav . Saib Hansen and Bowers (2008) rau kev txheeb xyuas kom tshuav nqi sib haum thiab Mutz and Pemantle (2015) rau kev txhawj xeeb txog tshuav nyiaj tshuav. Piv txwv li, kev siv nyiaj tshuav, Allcott (2011) pom muaj qee cov pov thawj uas tsis tau siv qhov kev xaiv kom raug rau hauv peb qhov Kev Ntsuam Xyuas Opower (saib cov lus 2; 2, 6, thiab 8). Rau lwm cov kev qhia, saib tshooj 21 ntawm Imbens and Rubin (2015) .
Lwm cov kev txhawj xeeb uas muaj feem xyuam nrog kev siv tau yog: (1) tsis muaj kev sib tw, qhov uas tsis yog txhua tus neeg hauv pawg kho mob tau txais kev kho mob, (2) ob tog neeg tsis ua haujlwm, tsis yog txhua tus neeg hauv pawg neeg kho mob tau txais kev kho mob thiab qee cov neeg Pawg tswj kav tau txais kev kho mob, (3) attrition, qhov twg qhov kev ntsuas tsis raug ntsuas rau qee cov neeg koom, thiab (4) cuam tshuam, qhov chaw kho mob yuav kis los ntawm cov neeg hauv kev kho mob rau cov neeg hauv kev tswj hwm. Saib tshooj 5, 6, 7, thiab 8 ntawm Gerber and Green (2012) ntxiv rau txhua qhov teeb meem no.
Yog xav paub ntxiv txog kev tsim nyog, saib Westen and Rosenthal (2003) , thiab ntxiv rau kev tsim kev siv nyob rau hauv cov ntaub ntawv loj, Lazer (2015) thiab tshooj 2 ntawm phau ntawv no.
Ib qho ntawm kev siv lwm tus sab nraud yog qhov chaw nyob rau hauv qhov kev pabcuam raug kuaj. Allcott (2015) muab kev xyuam xim theoretical thiab empirical kev kho kev xaiv ntawm qhov chaw xaiv. Qhov teeb meem no tseem tham txog Deaton (2010) . Lwm qhov ntawm kev siv lwm tus sab nraud yog seb lwm txoj haujlwm ua haujlwm ntawm tib qho kev pabcuam yuav muaj teebmeem zoo li ntawd. Hauv qhov no, kev sib piv ntawm Schultz et al. (2007) thiab Allcott (2011) qhia tau hais tias cov kev sib tw hu ua Opower tau muab coj los kho dua me dua cov kev sim los ntawm Schultz thiab cov npoj yaig (1.7% ntawm 5%). Allcott (2011) tsom xam tias cov kev sim ua raws li cov kev coj ua tau zoo dua vim yog txoj kev kho mob txawv: ib daim ntawv qhia txog kev nyeem ntawv yog ib feem ntawm txoj kev tshawb fawb los ntawm ib lub tsev kawm ntawv, piv rau cov ntawv luam tawm raws li ib feem ntawm ib pawg qhia los ntawm lub tuam txhab hluav taws xob.
- Kev kuaj mob ntawm kev kho mob (Seem 4.4.2)
Rau ib qho kev pom zoo ntawm heterogeneity ntawm kev kho teebmeem hauv kev sim ua haujlwm, saib tshooj 12 ntawm Gerber and Green (2012) . Rau kev qhia txog heterogeneity ntawm kev kho mob los ntawm kev kho mob, mus saib Kent and Hayward (2007) , Longford (1999) , thiab Kravitz, Duan, and Braslow (2004) . Kev xav txog kev heterogeneity ntawm kev kho mob qhov teeb meem feem ntau tsom xyuas rau qhov sib txawv raws li kev coj ua ntej kev kho mob. Yog tias koj nyiam heterogeneity raws li kev kho mob tom qab, ces yuav tsum siv ntau txoj hauv kev, xws li tus thawj xibfwb cov thawj coj (Frangakis and Rubin 2002) ; saib Page et al. (2015) rau kev ntsuam xyuas.
Coob tus neeg tshawb nrhiav kwv yees qhov kev sib txawv ntawm txoj kev kho mob uas siv cov kev hloov hauv txoj kev ua haujlwm, tab sis cov kev ua tshiab dua kev ntseeg siv tshuab; saib, piv txwv li, Green and Kern (2012) , Imai and Ratkovic (2013) , Taddy et al. (2016) , thiab Athey and Imbens (2016a) .
Muaj qee qhov kev tsis pom zoo txog kev tshawb nrhiav ntawm kev sib haum xeeb ntawm kev sib txawv vim tias muaj ntau qhov kev sib piv thiab "nuv ntses." Muaj ntau hom kev qhia uas tuaj yeem pab qhia txog kev txhawj xeeb txog ntau yam sib piv (Fink, McConnell, and Vollmer 2014; List, Shaikh, and Xu 2016) . Ib txoj kev txhawj xeeb txog "kev nuv ntses" yog kev tso npe ua ntej, uas pib ua rau kev xav txog psychology (Nosek and Lakens 2014) , kev tswjfwm (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , thiab kev lag luam (Olken 2015) .
Hauv kev tshawb fawb los ntawm Costa and Kahn (2013) tsuas yog kwv yees li ntawm ib nrab ntawm cov tsev neeg hauv qhov kev sim ua tau txuas rau cov lus qhia txog pej xeem. Cov neeg nyeem xav paub txog cov ntsiab lus no yuav tsum xa rau daim ntawv xub thawj.
- Mechanisms (seem 4.4.3)
Mechanisms yog qhov tseem ceeb heev, tab sis lawv tig los ua nyuaj rau txoj kev kawm. Kev tshawb xyuas txog cov mechanisms ze rau txoj kev kawm ntawm cov neeg tuaj kho hauv kev xav ntawm psychology (tab sis pom kuj VanderWeele (2009) rau kev sib piv ntawm ob lub tswv yim). Zauv tshwm tawm los nrhiav txoj haujlwm, xws li kev tsim tawm hauv Baron and Kenny (1986) , muaj ntau heev. Hmoov tsis, nws hloov tawm tias cov txheej txheem ntawm cov txheej txheem nyob ntawm seb muaj qee qhov kev xav (Bullock, Green, and Ha 2010) thiab raug kev txom nyem thaum muaj ntau ntau cov tswv yim, raws li ib tus yuav xav tau ntau zaus (Imai and Yamamoto 2013; VanderWeele and Vansteelandt 2014) . Imai et al. (2011) thiab Imai and Yamamoto (2013) muab qee txoj kev tawm tsam. Tsis tas li ntawd, VanderWeele (2015) muab cov ntawv kho mob ntev ntev nrog ntau cov ntsiab lus tseem ceeb, nrog rau kev txhim kho rau kev soj ntsuam.
Ib qho kev ua txawv cais ntawm cov kev sim uas sim muab cov mechanism ncaj qha (xws li, muab cov vitamin C sailors). Hmoov tsis, hauv ntau qhov chaw kawm txog kev sib raug zoo, muaj ntau lub tswv yim ntau yam thiab nws tsis yooj yim tsim cov kev kho mob uas hloov ib qho tsis hloov lwm tus. Qee cov kev qhia txog kev hloov kho cov tswv yim yog Imai, Tingley, and Yamamoto (2013) , Ludwig, Kling, and Mullainathan (2011) , thiab Pirlott and MacKinnon (2016) .
Cov kws tshawb fawb tau sim ua qhov kev sim siab tiag tiag yuav tsum muaj kev txhawj xeeb txog ntau hom kev ntsuas kev ntseeg siab; saib Fink, McConnell, and Vollmer (2014) thiab List, Shaikh, and Xu (2016) yog xav paub ntxiv.
Thaum kawg, cov mechanisms tseem muaj keeb kwm ntev nyob rau lub tswv yim ntawm kev tshawb fawb raws li piav qhia los ntawm Hedström and Ylikoski (2010) .
- Siv cov kev kawm uas twb muaj lawm (seem 4.5.1)
Yog xav paub ntxiv txog kev siv cov ntawv tshawb fawb thiab kev tshawb xyuas kev soj ntsuam los ntsuas kev ntxub ntxaug, saib Pager (2007) .
- Tsim koj tus kheej sim (seem 4.5.2)
Txoj kev uas yuav los ua rau cov neeg tuaj yeem tsim ua yog Amazon Mechanical Turk (MTurk). Vim hais tias MTurk cov haujlwm ntawm cov kev sim (lab) thwm nyiaj-them tib neeg kom ua tiav cov dej num uas lawv tsis ua rau cov neeg tshawb fawb dawb tau pib siv cov Turkers (cov neeg ua haujlwm rau MTurk) raws li cov neeg tuaj koom kev sim, ua rau cov ntaub ntawv ceev dua thiab pheej yig dua li ua tiav nyob rau hauv cov qauv kev sim hauv tsev kawm ntawv (Paolacci, Chandler, and Ipeirotis 2010; Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012; Rand 2012; Berinsky, Huber, and Lenz 2012) .
Feem ntau, qhov zoo tshaj plaws ntawm kev siv cov neeg koom rau hauv kev ua haujlwm ntawm MTurk yog logistical. Hos cov kev sim (lab) yuav siv sij hawm khiav mus rau qhov kev sib tw ua hauj lwm yuav ua rau lub hlis dhau los, kev sim nrog cov neeg ua haujlwm tau txais los ntawm MTurk tuaj yeem khiav hauv hnub. Piv txwv li, Berinsky, Huber, and Lenz (2012) tau txais 400 hom kawm hauv ib hnub nkaus rau kev koom tes hauv kev sim 8 feeb. Tsis tas li ntawd, cov neeg koom nrog no tuaj yeem nrhiav tau rau txhua lub hom phiaj (suav nrog kev tshawb fawb thiab kev sib koom ua ke, raws li tau hais tseg hauv tshooj 3 thiab 5). Qhov yooj yim ntawm recruitment txhais tau tias cov neeg tshawb xyuas yuav khiav cov theem ntawm cov kev sim uas muaj feem xyuam nrog kev ceev nrooj.
Ua ntej yuav tau koom tes ntawm MTurk rau koj qhov kev sim, koj muaj plaub yam tseem ceeb uas koj yuav tsum paub. Ua ntej, cov kws tshawb fawb tau muaj kev tsis sib txuam ntawm kev sim ntawm Turkers. Vim hais tias qhov kev tsis pom zoo no tsis meej li, nws yog ib qho nyuaj rau kev tawm tsam nrog cov pov thawj. Txawm li cas los xij, tom qab ob peb xyoo dhau los ntawm kev tshawb fawb siv Turkers, peb tuaj yeem txiav txim siab tias qhov kev tsis ntseeg no tsis yog raug cai. Muaj ntau txoj kev tshawb fawb tau muab piv rau cov pej xeem hauv Turkers nrog rau lwm cov neeg thiab ntau cov kev tshawb fawb tau muab piv rau cov kev sib tw nrog Turkers los ntawm lwm cov neeg. Ua txhua yam haujlwm no, kuv xav tias qhov zoo tshaj plaws rau koj xav txog nws yog tias cov Turkers yog ib qho qauv tsim nyog, zoo li cov menyuam kawm ntawv tab sis ntau ntau ntau hom (Berinsky, Huber, and Lenz 2012) . Yog li, raws li cov menyuam kawm ntawv yog ib haiv neeg tsim nyog rau qee yam, tab sis tsis yog txhua yam, kev tshawb fawb, Turkers yog cov pej xeem tsim nyog rau qee yam, tab sis tsis yog txhua yam, tshawb fawb. Yog hais tias koj yuav mus ua haujlwm nrog cov Turkers, ces nws ua kom paub nyeem ntau cov kev tshawb fawb no thiab to taub lawv cov nuances.
Thib ob, cov kws tshawb nrhiav tau tsim cov kev coj ua zoo tshaj plaws rau kev ua kom tiav ntawm MTurk cov kev sim, thiab koj yuav tsum kawm txog thiab ua raws li cov kev coj ua zoo tshaj plaws (Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012) . Piv txwv li, cov kws tshawb fawb siv cov Turkers txhawb kom siv cov kws tshawb fawb kom tshem cov neeg tsis koom nrog (Berinsky, Margolis, and Sances 2014, 2016) (tab sis saib kuj DJ Hauser and Schwarz (2015b) thiab DJ Hauser and Schwarz (2015a) ). Yog tias koj tsis tshem tawm cov neeg tsis tuaj yeem, ces muaj kev cuam tshuam ntawm txoj kev kho mob tuaj yeem ntxuav los ntawm lub suab nrov uas lawv qhia, thiab xyaum cov neeg tsis muaj kev koom nrog yuav ua tau ntau. Hauv kev sim ntawm Huber thiab cov npoj yaig (2012) , muaj li ntawm 30% ntawm cov neeg koom tsis tau cov kev ntsuam xyuas cov xim yooj yim. Lwm cov teeb meem uas feem ntau tshwm sim thaum cov neeg siv Turkers yog cov neeg tsis tuaj yeem (Chandler et al. 2015) thiab attrition (Zhou and Fishbach 2016) .
Qhov peb, sib piv rau lwm hom kev sim, Cov kev sim MTurk tsis tuaj yeem ntsuas; Stewart et al. (2015) kwv yees hais tias thaum twg los tau sijhawm tsuas muaj li ntawm 7,000 tus neeg ntawm MTurk.
Thaum kawg, koj yuav tsum paub tias MTurk yog ib lub zej zog nrog nws cov cai thiab cov cai (Mason and Suri 2012) . Nyob rau tib txoj kev uas koj yuav sim kom paub txog kev cai dab qhuas ntawm ib lub teb chaws uas koj yuav mus khiav koj cov kev sim, koj yuav tsum sim paub ntau ntxiv txog cov kab lis kev cai thiab cov qauv ntawm cov Turkers (Salehi et al. 2015) . Thiab koj yuav tsum paub tias cov Turkers yuav tham txog koj txoj kev sim yog tias koj ua tej yam tsis tsim nyog lossis tsis ncaj ncees (Gray et al. 2016) .
MTurk yog ib txoj hauv kev yooj yim heev rau kev nrhiav neeg koom rau hauv koj cov kev sim, seb lawv zoo li hom (lab), xws li ntawm Huber, Hill, and Lenz (2012) , los yog ntau tshaj qhov zoo li hauv, xws li cov Mason and Watts (2009) , Goldstein, McAfee, and Suri (2013) , Goldstein et al. (2014) , Horton and Zeckhauser (2016) , thiab Mao et al. (2016) .
- Tsim koj tus kheej cov khoom (seem 4.5.3)
Yog tias koj tab tom xav txog kev tsim koj tus kheej cov khoom, kuv xav kom koj nyeem cov lus qhia los ntawm MovieLens pawg hauv Harper and Konstan (2015) . Lub tswv yim tseem ceeb los ntawm lawv cov kev paub yog tias rau txhua qhov kev vam meej muaj ntau, ntau yam kev tsis txaus. Piv txwv, cov MovieLens pawg pib ua lwm yam khoom, xws li GopherAnswers, uas tau ua tiav tsis tiav (Harper and Konstan 2015) . Ib qho piv txwv ntawm ib tug neeg tshawb nrhiav poob thaum sim npaj tsim khoom yog Edward Castronova lub sim los tsim ib qho kev ua si hauv online hu ua Arden. Dua li ntawm $ 250,000 nyiaj txiag, qhov project yog ib qho chaw ntiav (Baker 2008) . Cov haujlwm zoo li GopherAnswers thiab Arden yog hmoov tsis ntau ntau dua li tej yaam num xws li MovieLens.
- Koom tes nrog cov neeg muaj hwj chim loj (Tshooj 4.5.4)
Kuv tau hnov lub tswv yim ntawm Pasteur Quadrant tau tham ntau zaus ntawm cov tuam txhab lag luam, thiab nws pab npaj cov kev tshawb nrhiav hauv Google (Spector, Norvig, and Petrov 2012) .
Kev sib tw thiab cov neeg ua haujlwm pabcuam txoj kev tshawb kawm (2012) tseem npaj siab los kuaj xyuas cov kev kho mob ntawm cov kev khomob rau cov phoojywg ntawm cov neeg uas tau txais lawv. Vim yog qhov tsim los ntawm kev sim, cov kev txeej no yog qhov nyuaj rau pom tseeb tseeb; cov nyiam nyeem ntawv yuav tsum pom Bond et al. (2012) rau kev sib tham sib ntxiv. Jones thiab cov npoj yaig (2017) kuj tau sim txoj kev sim zoo ib yam li thaum xyoo 2012 kev xaiv tsa. Cov kev sim no yog ib feem ntawm qhov kev coj ua ntawm kev sim nyob rau hauv kev tshawb fawb txog kev yaum kom txhawb kev pov npav (Green and Gerber 2015) . Cov kev sib tw tau tawm-kev-pov npav no feem ntau, vim yog lawv nyob hauv Pasteur Quadrant. Qhov ntawd yog, muaj ntau tus neeg uas tau mob siab ua rau kev xaiv tsa thiab kev pov npav tuaj yeem yog tus cwj pwm zoo rau kev ntsuam xyuas ntau qhov kev xav txog kev hloov tus cwj pwm thiab kev coj noj coj ua.
Kev tawm tswv yim txog kev sib tw ua haujlwm nrog cov koom haum sib txig xws li kev cai lij choj, tsoomfwv, thiab cov lag luam, saib Loewen, Rubenson, and Wantchekon (2010) , JA List (2011) , thiab Gueron (2002) . Rau kev xav txog kev sib koom tes nrog cov koom haum muaj peev xwm cuam tshuam kev tshawb fawb, saib King et al. (2007) thiab Green, Calfano, and Aronow (2014) . Kev koom tes kuj tuaj yeem ua rau cov lus nug raug cai, raws li tau hais los ntawm Humphreys (2015) thiab Nickerson and Hyde (2016) .
- Cov tswv yim tsim (ntu 4.6)
Yog tias koj yuav los tsim ib qho kev tshuaj ntsuam xyuas ua ntej yuav sim koj sim, kuv xav hais tias koj pib los ntawm kev nyeem cov txheej txheem qhia. Cov kev cai tswj hwm (Consolidated Standard Reporting of Trials) yog tsim los ntawm cov tshuaj (Schulz et al. 2010) thiab hloov kho rau kev tshawb fawb (Mayo-Wilson et al. 2013) . Ib cov txheej txheem txog kev qhia tau tsim los ntawm cov kws kho los ntawm Phau Tsom Fwm Kev Ntsuam Xyuas Kev Ntsej Muag Xeeb (Gerber et al. 2014) (saib Mutz and Pemantle (2015) thiab Gerber et al. (2015) ). Thaum kawg, cov txheej txheem qhia tau raug tsim los ntawm kev tshawb fawb (APA Working Group 2008) , thiab saib Simmons, Nelson, and Simonsohn (2011) .
Yog tias koj tsim ib qho kev tshuaj ntsuam xyuas, koj yuav tsum xav txog kev sau npe ua ntej vim tias pre-registration yuav ua rau kom muaj kev ntseeg siab uas lwm tus muaj nyob hauv koj cov txiaj ntsig. Tsis tas li, yog tias koj ua haujlwm nrog tus khub, nws yuav txo koj tus khub txoj kev muaj peev xwm los hloov qhov kev ntsuam xyuas tom qab pom cov kev tshwm sim. Kev sau npe ua ntej yog ua rau cov kev xav hauv psychology (Nosek and Lakens 2014) , kev tswjfwm txog science (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , thiab Economics (Olken 2015) .
Kev tsim tawm tswv yim tshwj xeeb rau kev sim kawm hauv online kuj muaj nyob rau hauv Konstan and Chen (2007) thiab Chen and Konstan (2015) .
Qhov kuv tau hu ua armada lub tswv yim qee zaus hu ua kev tshawb fawb ntawm qhov kev pab cuam ; saib Wilson, Aronson, and Carlsmith (2010) .
- Ua kom muaj pes tsawg tus nqi qis tshaj tawm (nqe lus 4.6.1)
Xav paub ntau ntxiv hauv MusicLab thwmsim, saib Salganik, Dodds, and Watts (2006) , Salganik and Watts (2008) , Salganik and Watts (2009b) , Salganik and Watts (2009a) , thiab Salganik (2007) . Yog xav paub ntxiv txog kev sib tw-tag nrho cov kev lag luam, saib Frank and Cook (1996) . Yog xav paub ntxiv txog untangling hmoov thiab kev txawj ntau dua, saib Mauboussin (2012) , Watts (2012) , thiab Frank (2016) .
Muaj lwm txoj kev los mus tshem tawm cov neeg them nqi uas cov kws tshawb fawb yuav tsum siv nrog kev ceev faj. Nyob rau hauv ntau qhov kev kawm hauv online thev naus laus zis yog cov uas tau muab coj los ua cov kev sim thiab yeej tsis them nyiaj. Piv txwv ntawm txoj kev qhia no muaj xws li Restivo thiab van de Rijt's (2012) sim rau kev vam hauv Wikipedia thiab Bond thiab tus neeg ua hauj lwm (2012) sim seb puas txhawb cov neeg los pov npav. Cov kev sim no tsis tshua muaj pes tsawg tus nqi-theej, lawv muaj pes tsawg tus nqi rau cov neeg tshawb xyuas . Nyob rau hauv cov kev sim no, txawm tias tus nqi rau txhua tus neeg tuaj yeem tsis tshua muaj me me, cov nqi qiv yuav ua tau ntau heev. Cov kws tshawb fawb pom kev sib txawv hauv cov kev sim hauv internet feem ntau pom tias qhov tseem ceeb ntawm txoj kev kho mob me me los ntawm kev hais tias cov teeb meem me no tseem ceeb thaum siv rau ntau tus neeg. Tib lub tswv yim tib yam siv rau cov nqi uas cov kws soj ntsuam cov neeg koom nrog. Yog tias koj txoj kev sim ua rau ib plhom leej neeg yuav tsum muab pov tseg ib feeb, qhov kev sim tsis yog teeb meem rau ib tus neeg twg, tab sis nyob rau hauv cov khoom sib xyaw nws tau khwv tau yuav luag ob xyoos.
Lwm txoj kev los tsim kom muaj pes tsawg tus nqi them rau cov neeg tuaj koom yog siv hom rho npe, ib qho uas tau siv los ntawm kev tshawb fawb (Halpern et al. 2011) . Yog xav paub ntxiv txog kev tsim cov neeg siv kev paub zoo, saib Toomim et al. (2011) . Xav paub ntxiv txog kev siv bots los tsim cov xoom pes tsawg tus nqi thwmsim saib ( ??? ) .
- Hloov, Kho, thiab Txo (Tshooj 4.6.2)
Qhov peb R's raws li Asmeskas npaj siab los ntawm Russell and Burch (1959) yog raws li nram no:
"Hloov txhais tau tias cov hloov rau meej pem nyob siab dua cov tsiaj ntawm insentient khoom. Yuav txo tau txhais tau tias yuav txo tau nyob rau hauv lub xov tooj ntawm cov tsiaj siv los muab cov ntaub ntawv ntawm ib tug muab npaum li cas thiab precision. Ca txhais tau tias tej txo nyob rau hauv cov xwm txheej los yog heev ntawm tsis tub kom cov txheej txheem siv rau cov tsiaj uas tseem muaj yuav tsum tau siv. "
Peb R cov lus uas kuv hais no tsis ua raws li cov ntsiab cai coj dawb huv uas tau piav qhia hauv tshooj 6. Xwb, lawv yog cov ntsiab lus tsim dua ib qho ntawm cov txheej txheem-kev pabcuam-tshwj xeeb ntawm qhov kev sim ntawm tib neeg kev sim.
Raws li ntawm thawj R ("hloov"), muab piv rau qhov kev xav ntawm kev sib txawv (Kramer, Guillory, and Hancock 2014) thiab qhov kev sim siab ntawm qhov kev xav tau (Lorenzo Coviello et al. 2014) muab qee cov lus qhia txog cov lag luam nyob hauv kev sim ntawm kev sim mus rau qhov kev sim (thiab lwm cov lej zoo li txuam nrog sim ua cov kev sim nyob rau hauv cov ntaub ntawv uas tsis yog kev sim; saib tshooj 2). Ntxiv nrog rau cov txiaj ntsig zoo, kev hloov ntawm kev sim mus rau cov kev kawm tsis yog kev sim kuj ua rau cov kws soj ntsuam los kawm cov kev kho mob uas lawv tsis muaj peev xwm muab xa mus. Cov kev coj zoo thiab cov kev pabcuam no tuaj yeem them tus nqi, tabsis. Cov kws tshawb fawb ntawm cov kws tshawb fawb tau tswj xyuas ntau yam xws li nrhiav neeg ua haujlwm, kev xaiv, thiab qhov kev kho mob. Piv txwv li, ib qho kev txwv ntawm kev dej nag los ua ib txoj kev kho yog tias nws ob leeg nce zoo thiab txo tsawg dua qub. Hauv kev sim kawm, txawm li cas los, Kramer thiab cov npoj yaig tau kho qhov zoo thiab qhov tsis zoo ntawm nws tus kheej. Cov kev qhia siv los ntawm Lorenzo Coviello et al. (2014) ntxiv tau ntxiv los ntawm L. Coviello, Fowler, and Franceschetti (2014) . Rau cov kev taw qhia rau cov txiaj ntsim hloov siab, uas yog txoj kev siv los ntawm Lorenzo Coviello et al. (2014) , saib Angrist and Pischke (2009) (tsawg dua qub) lossis Angrist, Imbens, and Rubin (1996) (ntau tshaj qub). Rau kev txhawj xeeb ntawm cov kev hloov pauv, saib Deaton (2010) , thiab rau kev taw qhia rau cov khoom siv suab paj nruag nrog cov cuab yeej tsis muaj zog (los nag yog lub cuab tam tsis muaj zog), saib Murray (2006) . Feem ntau, qhov kev qhia zoo tshaj plaws rau ntuj tsim yog muab los ntawm Dunning (2012) , thaum Rosenbaum (2002) , ( ??? ) , thiab Shadish, Cook, and Campbell (2001) muab tswv yim zoo txog kev kwv yees cov teebmeem tsis muaj kev sim.
Nyob rau hauv cov nqe lus ntawm tus thib ob R ("refinement"), muaj scientific thiab logistical trade-offs thaum xav tias yuav hloov tus tsim ntawm Emotional Contagion ntawm thaiv cov hlab cov posts rau boosting posts. Piv txwv, nws yuav yog cov ntaub ntawv uas kev siv ntawm Xovxwm Pub ua rau nws yog ib qho yooj yim ua ib qho kev sim ua qhov twg cov posts raug thaiv tsis yog ib qho uas lawv tau txhawb (nco ntsoov tias kev sim nrog blocking cov posts yuav tsum tau siv raws li ib txheej rau saum Xov Tooj Pub uas tsis muaj kev xav tau kev hloov ntawm lwm qhov system). Txoj kev xav, txawm li cas los, txoj kev xav los ntawm qhov kev sim tau tsis meej meej hais qhia ib qho qauv ntawm lwm qhov. Hmoov tsis, Kuv tsis paub ntau txog kev tshawb fawb txog kev zoo nkauj ntawm kev thaiv thiab txhawb cov ntsiab lus hauv Xov Tooj Pub. Tsis tas li ntawd, kuv tsis tau pom ntau txoj kev tshawb fawb txog kev kho kom tsis txhob ua kom lawv tsis muaj teeb meem; ib qho tshwj xeeb yog B. Jones and Feamster (2015) , uas yog txiav txim siab txog kev ntsuas ntawm Internet censorship (lub ntsiab lus kuv tham hauv tshooj 6 nyob rau hauv kev sib raug zoo rau txoj kev tshawb Encore (Burnett and Feamster 2015; Narayanan and Zevenbergen 2015) ).
Raws li qhov thib peb R ("txo qhov"), zoo introductions rau tsoos hluav taws xob kev ntsuam xyuas yog muab los ntawm Cohen (1988) (phau ntawv) thiab Cohen (1992) (tsab xov xwm), thaum Gelman and Carlin (2014) muab qhov txawv ntawm me ntsis. Cov kev kho mob ua ntej yuav muab tso rau hauv cov qauv tsim thiab tshawb xyuas cov kev sim; Tshooj 4 ntawm Gerber and Green (2012) muaj kev qhia zoo rau ob qho tib si, thiab Casella (2008) muab kev kho mob ntxiv. Cov tswv yim uas siv cov lus qhia txog kev mob ua ntej no nyob rau hauv kev xaiv yog feem ntau hu ua cov qauv kev sim los yog cov qauv kev sim ua qauv (cov ntsiab lus tsis raug siv thoob hauv cov zej zog); cov tswv yim no muaj feem xyuam nrog cov qauv kev xyaum ua qauv qhia nyob rau hauv tshooj 3. Saib Higgins, Sävje, and Sekhon (2016) ntxiv rau kev siv cov qauv no hauv cov kev sim ua loj. Cov kev kho mob ua ntej kev kho kuj tuaj yeem raug tso rau hauv kev soj ntsuam theem. McKenzie (2012) tshawb txog qhov sib txawv ntawm qhov sib txawv ntawm kev txheeb xyuas kev sim thev naus laus zis. Saib Carneiro, Lee, and Wilhelm (2016) ntxiv rau cov lag luam tawm ntawm ntau txoj kev mus nce rau qhov kev txiav txim ntawm kev kho mob. Thaum kawg, thaum txiav txim siab tias yuav ua tiag nrog cov kev kho mob ua ntej kev kho los ntawm cov qauv tsim (los yog ob qho tib si), muaj ob peb yam los xav txog. Nyob rau hauv qhov chaw uas cov kws tshawb fawb xav ua kom pom tias lawv tsis yog "nuv ntses" (Humphreys, Sierra, and Windt 2013) , siv cov kev kho mob ua ntej kho nyob rau hauv cov qauv tsim muaj peev xwm yuav pab tau (Higgins, Sävje, and Sekhon 2016) . Nyob rau lub sijhawm uas cov neeg tuaj koom nrog ua ke, tshwjxeeb hauv kev kawm online, tshwj xeeb tshaj yog siv cov ntaub ntawv qhia ua ntej hauv cov qauv tsim tawm yuav nyuaj rau logistically; saib, piv txwv, Xie and Aurisset (2016) .
Nws tsim nyog ntxiv qhov me ntsis ntawm qhov kev xav txog vim li cas ib qho kev sib txawv ntawm kev sib txawv yuav ua tau zoo dua li qhov sib txawv ntawm-hauv-txhais tau tias ib qho. Muaj ntau cov txiaj ntsim hauv online muaj cov variance ntau heev (saib xws li, RA Lewis and Rao (2015) thiab Lamb et al. (2015) ) thiab tsis tshua muaj sij hawm ntev. Hauv qhov no, qhov qhab nia hloov pauv yuav muaj feem me me, qhov nce lub zog ntawm qhov xeem tawm. Ib qho laj thawj no tsis yog siv ntau dua yog tias ua ntej hnub nyoog digital, nws tsis yog hom kev kho mob ua ntej. Ntau txoj kev xav txog qhov no yog xav txog kev sim kom ntsuas seb qhov kev qoj ib ce ua rau poob phaus. Yog tias koj txais qhov kev sib txawv hauv-txhais tau tias, koj qhov kev kwv yees yuav muaj kev sib txawv tshwm sim los ntawm kev sib txawv hauv kev lag luam hauv cov neeg nyob. Yog tias koj ua qhov txawv ntawm qhov sib txawv, txawm li cas los xij, qhov kev hloov ntawm qhov txawv ntawm qhov hnyav yog muab tshem tawm, thiab koj tuaj yeem tshawb xyuas qhov txawv ntawm qhov kev kho mob.
Thaum kawg, kuv xav muab qhov thib plaub R: "repurpose". Qhov no yog, yog tias cov neeg tshawb xyuas pom lawv tus kheej nrog cov ntaub ntawv ntau dua li qhov lawv xav tau los hais lawv cov lus nug, lawv yuav tsum rov qab ua cov ntaub ntawv los nug cov lus nug tshiab. Piv txwv, xav tias Kramer thiab cov npoj yaig tau siv qhov kev sib txawv ntawm kev sib txawv thiab pom lawv tus kheej nrog ntau cov ntaub ntawv tshaj qhov lawv xav tau los hais txog lawv cov lus nug. Ntau dua li tsis siv cov ntaub ntawv rau tag nrho qhov twg, lawv yuav tau kawm txog qhov loj ntawm cov nyhuv ua lub luag haujlwm ntawm kev kho mob ua ntej kev xav. Ib yam li Schultz et al. (2007) pom tias cov kev kho mob zoo txawv rau cov neeg siv lub teeb thiab hnyav, tej zaum cov teebmeem ntawm Xovtooj Pub tau sib txawv rau cov neeg uas twb tau xa mus rau cov ntaub ntawv zoo siab (lossis tu siab). Rov ua dua tuaj yeem ua rau "nuv ntses" (Humphreys, Sierra, and Windt 2013) thiab "p-hacking" (Simmons, Nelson, and Simonsohn 2011) , tab sis cov no yog cov muaj kev ceeb toom nrog kev qhia ncaj ncees (Simmons, Nelson, and Simonsohn 2011) , pre-registration (Humphreys, Sierra, and Windt 2013) , thiab kev siv tshuab txoj kev kawm uas sim kom tsis txhob muaj phom sij ntxiv.
