ntxiv tswv yim

Qhov no txhais tau tsim los ntawm ib tug computer. ×

You are reading the Open Review Edition of Bit by Bit. Click here to read the 1st Edition.

ntxiv tswv yim

Seem no yog tsim los siv raws li ib tug siv, es tsis yuav tsum tau nyeem raws li ib tug narrative.

Introduction (Section 4.1)

Yog muaj lus nug txog causality nyob rau hauv kev sib raug zoo kev tshawb fawb yog feem ntau complex thiab intricate. Rau ib ntsiab mus kom ze rau causality raws li causal graphs, saib Pearl (2009) , thiab rau ib tug ntsiab mus kom ze raws li tej zaum tau, saib Imbens and Rubin (2015) (thiab cov kev appendix nyob rau hauv tshooj no). Rau ib tug sib piv ntawm cov ob le caag, saib Morgan and Winship (2014) . Rau ib tug formal mus kom ze rau defining ib confounder, saib VanderWeele and Shpitser (2013) .

Nyob rau hauv tshooj, kuv tsim dab tsi zoo li ib tug kaj kab ntawm peb muaj peev xwm ua causal kev kwv yees los ntawm kev sim thiab uas tsis yog-xyaum ua tej yam ntaub ntawv. Nyob rau hauv kev muaj tiag, kuv xav hais tias cov txim hluav taws xob yog blurrier. Piv txwv li, txhua leej txhua tus lees txais tias haus luam yeeb ua rau mob cancer txawm tias peb yeej tsis tau ua ib tug xaiv tswj xyaum ua tej yam uas forces neeg haus luam yeeb. Rau yav tom ntej hauv phau ntawv ntev kev kho mob rau ua causal kev kwv yees los ntawm cov-cov ntaub ntawv saib Rosenbaum (2002) , Rosenbaum (2009) , Shadish, Cook, and Campbell (2001) , thiab Dunning (2012) .

Tshooj 1 thiab 2 ntawm Freedman, Pisani, and Purves (2007) muaj ib tug meej kev taw qhia mus rau hauv qhov sib txawv ntawm thwmsim, tswj thwmsim, thiab xaiv tswj thwmsim.

Manzi (2012) muaj ib tug fascinating thiab tshwmsim kev taw qhia mus rau hauv lub philosophical thiab statistical underpinnings ntawm xaiv tswj thwmsim. Nws kuj muaj nthuav tiag tiag lub ntiaj teb piv txwv ntawm lub hwj chim ntawm kws nyob rau hauv lub lag luam.

Yuav ua li cas yog kev? (Section 4.2)

Casella (2008) , Box, Hunter, and Hunter (2005) , Athey and Imbens (2016b) muab zoo qhia rau cov statistical yam ntawm kev sim tsim thiab tsom xam. Ntxiv mus, muaj cov zoo heev kev kho mob ntawm kev siv ntawm thwmsim nyob rau hauv ntau txawv teb: economics (Bardsley et al. 2009) , Sociology (Willer and Walker 2007; Jackson and Cox 2013) , psychology (Aronson et al. 1989) , political science (Morton and Williams 2010) , thiab kev sib raug zoo txoj cai (Glennerster and Takavarasha 2013) .

Qhov tseem ceeb ntawm koom recruitment (eg, piv txwv) yog feem ntau nyob rau hauv-menyuam nyob rau hauv kev sim kev tshawb fawb. Txawm li cas los, yog hais tias tus nyhuv ntawm cov kev kho mob yog ua kev nyob rau hauv cov pejxeem, ces piv txwv yog ib qho tseem ceeb. Longford (1999) ua no taw tes kom meej meej thaum nws pab rau kev soj ntsuam xav txog thwmsim raws li ib tug pejxeem daim ntawv ntsuam xyuas nrog haphazard zauv.

Ob qhov ntev ntawm thwmsim: lab-teb thiab analog-digital (Section 4.3)

Lub dichotomy tias kuv hais ntawm lub chaw kuaj thiab teb thwmsim yog ib tug me ntsis yooj yim zog. Nyob rau hauv qhov tseeb, lwm yam kev soj ntsuam tau npaj ntau ncauj lus kom ntxaws typologies, nyob rau hauv particular uas cais cov ntau yam ntaub ntawv ntawm daim teb thwmsim (Harrison and List 2004; Charness, Gneezy, and Kuhn 2013) . Ntxiv mus, muaj cov ob tug lwm hom kev ua los ntawm kev sib raug zoo zaum uas tsis haum ntxiab mus rau hauv lub lab thiab teb dichotomy: daim ntawv ntsuam xyuas thwmsim thiab kev thwmsim. Survey thwmsim yog thwmsim siv cov infrastructure ntawm uas twb muaj lawm cov ntawv tshawb fawb thiab muab piv rau cov lus teb mus rau lwm txoj versions ntawm lub tib cov lus nug (ib co daim ntawv ntsuam xyuas thwmsim uas hais nyob rau hauv Tshooj 3); rau ntxiv rau daim ntawv ntsuam xyuas thwmsim saib Mutz (2011) . Social thwmsim yog thwmsim qhov twg cov kev kho mob yog ib co kev cai uas yuav tsuas muab los siv los ntawm ib tug tsoom fwv. Social thwmsim yog ze ze rau kev ntsuam xyuas. Yog xav paub ntxiv nyob rau hauv txoj cai thwmsim, saib Orr (1998) , Glennerster and Takavarasha (2013) , thiab Heckman and Smith (1995) .

Ib tug xov tooj ntawm cov ntaub ntawv tau muab piv lab thiab teb thwmsim nyob rau hauv cov abstract (Falk and Heckman 2009; Cialdini 2009) thiab nyob rau hauv cov nqe lus ntawm kev tshwm sim ntawm tej kev nyob rau hauv nom tswv science (Coppock and Green 2015) , economics (Levitt and List 2007a; Levitt and List 2007b; Camerer 2011; Al-Ubaydli and List 2013) thiab psychology (Mitchell 2012) . Jerit, Barabas, and Clifford (2013) muaj ib tug zoo kev tshawb fawb tsim rau muab piv tau los ntawm lub chaw kuaj thiab teb thwmsim.

Kev txhawj xeeb txog cov neeg hloov lawv tus cwj pwm vim hais tias lawv paub hais tias lawv yog raug zoo cai yog tej zaum hu ua kev thov los, thiab lawv tau raug kawm nyob rau hauv psychology (Orne 1962) thiab kev lag luam (Zizzo 2009) . Txawm tias feem ntau txuam nrog lab thwmsim, cov tib teeb meem yuav ua teeb meem rau teb thwmsim zoo li. Nyob rau hauv qhov tseeb, thov los yog tej thaum kuj hu ua Hawthorne los, ib lub sij hawm uas derives los ntawm ib daim teb xyaum ua tej yam, uas lub npe nrov illumination thwmsim tias pib nyob rau hauv 1924 nyob rau Hawthorne Works of lub Western Electric Company (Adair 1984; Levitt and List 2011) . Ob leeg thov los thiab Hawthorn los yog ze ze rau lub tswv yim ntawm reactive ntsuas sib tham nyob rau hauv Tshooj 2 (kuj saib Webb et al. (1966) ).

Lub keeb kwm ntawm teb thwmsim tau piav nyob rau hauv kev lag luam (Levitt and List 2009) , political science (Green and Gerber 2003; Druckman et al. 2006; Druckman and Lupia 2012) , psychology (Shadish 2002) , thiab pej xeem txoj cai (Shadish and Cook 2009) . Ib thaj tsam ntawm social science nyob qhov twg teb thwmsim sai sai los ua mas yog thoob ntiaj teb kev loj hlob. Rau ib tug zoo kev ntsuam xyuas ntawm uas ua hauj lwm tsis pub dhau economics saib Banerjee and Duflo (2009) , thiab rau ib tug tseem ceeb soj ntsuam saib Deaton (2010) . Rau ib tug saib xyuas ntawm no ua hauj lwm nyob rau hauv nom tswv science saib Humphreys and Weinstein (2009) . Thaum kawg, qhov kev zoo txoj kev sib tw muab kev koom tes nrog teb thwmsim tau tshawb nyob rau hauv nom tswv science (Humphreys 2015; Desposato 2016b) thiab kev loj hlob kev lag luam (Baele 2013) .

Nyob rau hauv tshooj, kuv pom tias pre-kev kho mob cov lus qhia yuav siv tau los txhim kho lub precision ntawm kwv yees li kev kho mob los, tab sis muaj ib co sib cav tswv yim hais txog qhov no mus kom ze: Freedman (2008) , Lin (2013) , thiab Berk et al. (2013) ; saib Bloniarz et al. (2016) kom paub ntau ntxiv.

Tsiv tshaj yooj yooj yim thwmsim (Section 4.4)

Kuv twb xaiv tau los ua kom pom tseeb rau ntawm peb lub tswv yim: validity, heterogeneity ntawm kev kho mob los, thiab mechanisms. Cov tswv yim muaj ntau lub npe nyob rau hauv txawv teb. Piv txwv li, psychologists yuav dhau mus yooj yim thwmsim los ntawm kev sib kho thiab moderators (Baron and Kenny 1986) . Lub tswv yim ntawm kev sib kho yuav yog dab tsi kuv hu mechanisms, thiab lub tswv yim ntawm cov moderators yuav yog dab tsi kuv hu lwm validity (eg, yuav cov ntsiab ntawm cov xyaum ua tej yam yuav txawv yog hais tias nws twb khiav nyob rau hauv tej yam) thiab heterogeneity ntawm kev kho mob los ( xws li, yog cov teebmeem loj rau ib txhia neeg dua li lwm cov neeg).

Lub xyaum ua tej yam ntawm Schultz et al. (2007) qhia tau hais tias yuav ua li cas kev sib raug zoo theories yuav siv tau los tsim zoo pab. Rau ib tug ntau general sib cav txog lub luag hauj lwm ntawm kev tshawb xav nyob rau hauv tsim zoo tiv thaiv, saib Walton (2014) .

Kev siv tau (Section 4.4.1)

Lub tswv yim ntawm kev nrog thiab lwm validity twb xub nkag tau rau hauv Campbell (1957) . Saib Shadish, Cook, and Campbell (2001) rau ib tug ntau ncauj lus kom ntxaws keeb kwm thiab ib tug faj elaboration ntawm statistical xaus validity, nrog validity, txua validity, thiab lwm validity.

Rau ib tug txheej txheem cej luam txog tej teeb meem muaj feem xyuam rau statistical xaus validity nyob rau hauv sim kawm saib Gerber and Green (2012) (rau ib tug social science foundations) thiab Imbens and Rubin (2015) (rau ib tug statistical foundations). Tej teeb meem ntawm statistical xaus validity uas tshwm sim uas nyob rau hauv online teb thwmsim muaj xws li tej teeb meem xws li computationally npaum txoj kev tsim cog qoob loo Team sib nrog nyob cov ntaub ntawv (Bakshy and Eckles 2013) .

Internal validity yuav tsis yooj yim los xyuas kom meej nyob rau hauv txoj teb thwmsim. Saib, piv txwv li, Gerber and Green (2000) , Imai (2005) , thiab Gerber and Green (2005) rau sib cav tswv yim hais txog qhov kev siv ntawm ib tug complex teb xyaum ua tej yam txog pov npav. Kohavi et al. (2012) thiab Kohavi et al. (2013) muab ib qho kev taw qhia mus rau hauv cov kev sib tw ntawm luv validity nyob rau hauv online teb thwmsim.

Ib tug loj kev txhawj xeeb nrog nrog validity yog muaj teeb meem nrog randomization. Ib txoj kev uas yuav muaj feem ntes teeb meem nrog rau cov randomization yog los sib piv cov kev kho mob thiab kev tswj pab pawg rau observable sijhawm. Qhov no zoo ntawm kev sib piv no yog hu ua ib tug tshuav nyiaj li cas daim tshev. Saib Hansen and Bowers (2008) rau ib tug statistical mus kom ze sib npaug tshev, thiab saib Mutz and Pemantle (2015) rau kev txhawj xeeb txog tshuav nyiaj li cas cov tshev mis. Piv txwv li, siv ib tug tshuav nyiaj li cas mus saib Allcott (2011) pom hais tias muaj yog ib co pov thawj uas lub randomization twb tsis siv kom thwj nyob rau hauv peb ntawm lub thwmsim nyob rau hauv ib co ntawm cov OPower thwmsim (saib Table 2; qhov chaw 2, 6, thiab 8). Rau lwm tus le caag, saib Imbens and Rubin (2015) , Tshooj 21.

Lwm yam loj tej kev txhawj xeeb muaj feem xyuam rau sab hauv validity yog: 1) ib tug-tog uas tsis yog raws kev cai, qhov twg tsis txhua leej txhua tus nyob rau hauv cov kev kho mob pab pawg neeg ua tau txais cov kev kho mob, 2) ob tog uas tsis yog raws kev cai, qhov twg tsis txhua leej txhua tus nyob rau hauv cov kev kho mob pab pawg neeg tau txais cov kev kho mob thiab ib co cov neeg nyob rau hauv lub pab pawg neeg tswj tau txais cov kev kho mob, 3) attrition, qhov twg ua tau yog tsis ntsuas rau ib co neeg koom, thiab 4) cuam, qhov twg cov kev kho mob nchuav dua los ntawm cov neeg nyob rau hauv cov kev kho mob mob rau cov neeg nyob rau hauv cov kev tswj kev mob. Saib Gerber and Green (2012) tshooj 5, 6, 7, thiab 8 rau ntau rau txhua tus ntawm cov teeb meem.

Yog xav paub ntxiv nyob rau hauv dlaim validity, saib Westen and Rosenthal (2003) , thiab rau ntau rau dlaim validity nyob rau hauv loj cov ntaub ntawv qhov chaw, Lazer (2015) thiab Tshooj 2 ntawm phau ntawv no.

Ib tug nam ntawm sab nraud validity yog qhov chaw uas ib tug pab yog mus soj ntsuam. Allcott (2015) muaj ib tug faj theoretical thiab empirical kev kho mob ntawm qhov chaw xaiv kev tsis ncaj ncees. Qhov no qhov teeb meem yog tseem tham txog nyob rau hauv Deaton (2010) . Nyob rau hauv tas li ntawd mus ua replicated nyob rau hauv ntau qhov chaw, lub tsev Zog Daim ntawv qhia txog kev pab kuj tau ntawm nws tus kheej kawm los ntawm ntau yam kev tshawb fawb pawg (xws li, Ayres, Raseman, and Shih (2013) ).

Heterogeneity ntawm kev kho mob los (Section 4.4.2)

Rau ib tug zoo heev txheej txheem cej luam ntawm heterogeneity ntawm kev kho mob los nyob rau hauv teb thwmsim, saib Tshooj 12 ntawm Gerber and Green (2012) . Rau qhia rau heterogeneity ntawm kev kho mob los nyob rau hauv kev kho mob nyuaj siab, saib Kent and Hayward (2007) , Longford (1999) , thiab Kravitz, Duan, and Braslow (2004) . Heterogeneity ntawm kev kho mob los feem ntau siab rau sib txawv raws li pre-kev kho mob yam ntxwv. Yog hais tias koj xav nyob rau hauv heterogeneity raws li post-kev kho mob tshwm sim, ces ntau approachs uas yuav tsum tau xws li tus thawj xib fwb stratification (Frangakis and Rubin 2002) ; saib Page et al. (2015) rau ib tug saib xyuas.

Muaj ntau soj ntsuam ntawm kwv yees lub heterogeneity ntawm kev kho mob los siv linear regression, tab sis newer txoj kev cia siab rau lub tshuab kev kawm, piv txwv li Green and Kern (2012) , Imai and Ratkovic (2013) , Taddy et al. (2016) , thiab Athey and Imbens (2016a) .

Muaj yog ib co tsis ntseeg tau pib hais txog kev tshawb pom ntawm heterogeneity ntawm los vim hais tias ntawm ntau yam sib piv cov teeb meem thiab "nuv ntses." Muaj ib tug ntau yam ntawm statistical le caag uas yuav pab tau qhov chaw nyob kev txhawj xeeb txog ntau yam kev sib piv (Fink, McConnell, and Vollmer 2014; List, Shaikh, and Xu 2016) . Ib tug mus kom ze rau kev txhawj xeeb txog "nuv ntses" yog pre-sau npe, uas yog ua nce ntau nyob rau hauv psychology (Nosek and Lakens 2014) , political science (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , thiab kev lag luam (Olken 2015) .

Nyob rau hauv txoj kev tshawb no ntawm Costa and Kahn (2013) tsuas hais txog ib nrab ntawm cov tsev neeg nyob rau hauv lub sim twb tau mus yuav txuas mus rau lub demographic ntaub ntawv. Txawj nyeem ntawv xav nyob rau hauv cov lus thiab tau muaj teeb meem nrog no tsom xam yuav tsum xa mus rau tus thawj daim ntawv.

Mechanisms (Section 4.4.3)

Mechanisms yog incredibly ib qho tseem ceeb, tab sis lawv tig mus ua nyuaj heev rau kawm. Kev tshawb fawb txog mechanisms zoo txog rau txoj kev tshawb no ntawm kev sib kho nyob rau hauv psychology (tab sis kuj saib VanderWeele (2009) rau ib tug leej kev sib piv ntawm ob lub tswv yim). Statistical le caag los nrhiav mechanisms, xws li qhov mus kom ze tsim nyob rau hauv Baron and Kenny (1986) , yog heev. Tu siab, nws hloov tawm hais tias cov neeg txheej txheem nyob ntawm seb ib co muaj zog cov kev xav (Bullock, Green, and Ha 2010) thiab raug kev txom nyem thaum muaj ntau yam mechanisms, raws li ib tug tej zaum xav tias yuav muaj ntau zaus (Imai and Yamamoto 2013; VanderWeele and Vansteelandt 2014) . Imai et al. (2011) thiab Imai and Yamamoto (2013) muaj ib co paub statistical txoj kev. Ntxiv mus, VanderWeele (2015) muaj ib phau ntawv-ntev kev kho mob nrog ib tug xov tooj ntawm ib qho tseem ceeb tau, xws li ib tug kev mus kom ze rau rhiab heev tsom xam.

Ib tug cais mus kom ze tsom rua kev uas sim muab lub mechanism ncaj qha (xws li, muab sailors vitamin C). Tu siab, nyob rau hauv ntau social science chaw muaj ntau feem ntau ntau yam mechanisms thiab nws yog ib qho nyuaj mus tsim kev kho mob uas hloov ib tug tsis muaj kev hloov rau lwm tus. Ib txhia le caag los experimentally Process mechanisms yog piav nyob rau hauv Imai, Tingley, and Yamamoto (2013) , Ludwig, Kling, and Mullainathan (2011) , thiab Pirlott and MacKinnon (2016) .

Thaum kawg, mechanisms kuj muaj ib tug ntev li keeb kwm nyob rau hauv lub philosophy ntawm science raws li piav los ntawm Hedström and Ylikoski (2010) .

Siv uas twb muaj lawm kev kawm (Section 4.5.1.1)

Yog xav paub ntxiv nyob rau kev siv ntawm xov xwm kev tshawb fawb thiab tshawb tshawb fawb los ntsuas kev ntxub ntxaug saib Pager (2007) .

Tsim kom muaj rau koj tus kheej xyaum ua tej yam (Section 4.5.1.2)

Feem ntau txoj kev mus nrhiav neeg koom rau kev hais tias koj tsim yog Amazon Neeg kho tshuab Turk (MTurk). Vim hais tias MTurk uas sib nrauj ntawm tsoos lab thwmsim-them cov neeg mus ua kom tiav kev pab raws qib hais tias lawv yuav tsis ua rau free-ntau soj ntsuam ntawm muaj twb pib siv Turkers (cov neeg ua hauj lwm rau MTurk) raws li cov neeg nyob rau hauv tib neeg kev kawm thwmsim ua nyob rau hauv ceev thiab pheej yig dua cov ntaub ntawv sau dua tsoos on-tsev kawm ntawv kuaj thwmsim (Paolacci, Chandler, and Ipeirotis 2010; Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012; Rand 2012; Berinsky, Huber, and Lenz 2012) .

Qhov loj tshaj lub zog ntawm thwmsim nrog koom recruited los ntawm MTurk yog logistical: lawv tso cai rau kev soj ntsuam kom txhij koom sai sai thiab raws li xav tau. Whereas lab thwmsim yuav coj lub lis piam mus khiav thiab teb thwmsim yuav siv lub hlis mus teev-up, thwmsim nrog koom recruited los ntawm MTurk yuav khiav nyob rau hauv hnub. Piv txwv li, Berinsky, Huber, and Lenz (2012) twb tau kom txhij 400 kawm nyob rau hauv ib hnub twg mus koom nyob rau hauv ib tug 8 feeb xyaum ua tej yam. Ntxiv mus, cov neeg koom yuav tsum recruited rau zoo yam lub hom phiaj (xws li cov ntawv tshawb fawb thiab loj kev sib koom tes, raws li sib tham nyob rau hauv tshooj 3 thiab 5). Qhov no yooj yim kev nrhiav neeg txhais tau tias kev soj ntsuam ntawm muaj peev xwm khiav sequences ntawm lwm yam kev nyob rau hauv sai succession.

Ua ntej recruiting neeg koom los ntawm MTurk rau koj tus kheej thwmsim, muaj plaub yam tseem ceeb los paub. Ua ntej, muaj ntau soj ntsuam ntawm muaj ib tug uas tsis yog-kev tsis ntseeg tau pib ntawm thwmsim uas Turkers. Vim hais tias qhov no tsis ntseeg tau pib tsis yog ib qho, nws yog ib qho nyuaj rau counter nrog pov thawj. Txawm li cas los, tom qab ob peb xyoos ntawm kev tshawb fawb uas siv cov Turkers, peb yuav tam sim no xaus hais tias qhov no tsis ntseeg tau pib tsis yog tshwj xeeb tshaj yog tsim nyog. Muaj ntau cov kev tshawb fawb muab piv rau cov demographics ntawm Turkers mus rau lwm cov coob thiab ntau cov kev tshawb fawb muab piv tau ntawm thwmsim nrog Turkers rau kev soj ntsuam los ntawm lwm yam coob. Muab tag nrho cov hauj lwm no, kuv xav hais tias qhov zoo tshaj plaws txoj kev rau koj xav txog nws ntawd yog Turkers yog ib tug tsim nyog yooj yim qauv, npaum li cov tub ntxhais kawm tab sis me ntsis ntau ntau haiv neeg (Berinsky, Huber, and Lenz 2012) . Yog li, cia li raws li cov menyuam kawm ntawv ib tug tsim nyog pejxeem rau ib co tab sis tsis yog txhua txhua sim kev tshawb fawb, Turkers yog ib tug tsim nyog pejxeem rau ib co tab sis tsis yog txhua txhua cov kev tshawb fawb. Yog hais tias koj yeej yuav tau mus ua hauj lwm nrog Turkers, ces nws yuav ua rau kev txiav txim zoo nyeem tau ntau ntau ntawm cov comparative kev tshawb fawb thiab to taub txog lawv nuances.

Ob, soj ntsuam ntawm muaj tsim zoo tshaj plaws-kev coj rau kom nrog validity ntawm Turk thwmsim, thiab koj yuav tsum paub txog thiab ua raws li cov zoo tshaj plaws-kev xyaum (Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012) . Piv txwv li, soj ntsuam siv Turkers kom siv screeners kom tshem tawm inattentive koom (Berinsky, Margolis, and Sances 2014; Berinsky, Margolis, and Sances 2016) (tiam sis kuj saib DJ Hauser and Schwarz (2015b) thiab DJ Hauser and Schwarz (2015a) ). Yog hais tias koj tsis tshem tawm inattentive koom, ces tej nyhuv ntawm cov kev kho mob yuav tsum ntxuav tawm los ntawm cov suab nrov ua tswvcuab los ntawm inattentive koom, thiab nyob rau hauv kev xyaum lub xov tooj ntawm inattentive koom yuav ua tau ntau yam pauv loj. Nyob rau hauv lub sim ntawm Huber thiab lug txhawb cov miv (2012) txog 30% ntawm cov neeg ua tsis tau tejyam yooj yim xim screeners. Ib qho teeb meem ntau nrog Turkers yog uas tsis yog-naïve koom (Chandler et al. 2015) .

Peb, txheeb ze rau ib co lwm yam ntaub ntawv ntawm cov thwmsim, MTurk thwmsim yuav tsis scale; Stewart et al. (2015) kwv yees tias thaum twg muab lub sij hawm muaj tsuas yog hais txog 7,000 tus neeg nyob rau MTurk.

Thaum kawg, koj yuav tsum paub tias MTurk yog ib lub zej zog nrog nws cov kev cai thiab kev cai (Mason and Suri 2012) . Nyob rau hauv tib txoj kev uas koj yuav sim mus nrhiav kom paub txog qhov kab lis kev cai ntawm ib lub teb chaws qhov chaw uas koj tau mus khiav koj cov thwmsim, koj yuav tsum tau sim kom paub ntau ntxiv txog cov kab lis kev cai thiab kev cai ntawm Turkers (Salehi et al. 2015) . Thiab, koj yuav tsum paub hais tias cov Turkers yuav tham tsis tau txog koj xyaum ua tej yam yog tias koj ua ib yam dab tsi tsis tsim nyog los yog unethical (Gray et al. 2016) .

MTurk yog ib tug heev kawg yooj yim txoj kev uas yuav nrhiav neeg koom rau koj thwmsim, seb puas muaj lab-li, xws li Huber, Hill, and Lenz (2012) , los yog ntau tshaj teb zoo li, xws li Mason and Watts (2009) , Goldstein, McAfee, and Suri (2013) , Goldstein et al. (2014) , Horton and Zeckhauser (2016) , thiab Mao et al. (2016) .

Tsim kom muaj rau koj tus kheej cov khoom (Section 4.5.1.3)

Yog hais tias koj xav sim los tsim koj tus kheej cov khoom, kuv pom zoo kom hais tias koj nyeem cov tswv yim uas muab los ntawm lub MovieLens pab pawg neeg nyob rau hauv Harper and Konstan (2015) . Ib tug tseem ceeb kev pom los ntawm lawv cov kev yog hais tias txhua txhua kev vam meej project muaj ntau ntau yam, muaj ntau yam failures. Piv txwv li, lub MovieLens pab pawg neeg launched lwm yam khoom xws li GopherAnswers uas ua tiav failures (Harper and Konstan 2015) . Lwm cov piv txwv ntawm ib tug soj ntsuam tsis thaum sim mus tsim ib yam khoom yog Edward Castronova sim los tsim kom tau ib tug nyob hauv internet ua si hu ua Arden. Txawm tias $ 250,000 nyob rau hauv cov nyiaj, lub project yog ib tug ntog (Baker 2008) . Tej yaam num xws li GopherAnswers thiab Arden yog hmoov tsis ntau ntau dua tej yaam num xws li MovieLens. Thaum kawg, thaum kuv hais tias kuv tsis paub txog lwm yam kev soj ntsuam uas tau ntse ua cov khoom rau rov experimentation ntawm no yog kuv kev: 1) koom siv cov khoom yog vim li cas nws muab lawv (xws li, lawv tsis tau them thiab lawv tsis tuaj pab dawb pab science) thiab 2) cov khoom tau tau muab siv rau ntau tshaj ib cov xyaum ua tej yam (ie, tsis yog tib yam xyaum ua tej yam ntau lub sij hawm uas sib txawv koom pas). Yog hais tias koj paub txog lwm cov piv txwv, thov qhia rau kuv paub.

Partner nrog cov haib (Section 4.5.2)

Kuv yeej hnov cov tswv yim ntawm Pasteur lub Quadrant sib tham nquag ntawm tech tuam txhab uas muag, thiab nws yuav pab npaj kev tshawb fawb kev hauv Google (Spector, Norvig, and Petrov 2012) .

Daim ntawv cog lus thiab lug txhawb cov miv 'txoj kev tshawb no (2012) tseem npaj siab mus ntes cov nyhuv ntawm cov kev kho mob rau cov phooj ywg ntawm cov neeg uas tau txais lawv. Vim hais tias ntawm cov qauv ntawm qhov xyaum ua tej, cov spillovers yog nyuaj mus ntes cleanly; xav nyeem yuav tsum saib Bond et al. (2012) rau ib tug ntau sib tham. Qhov no xyaum ua tej yam yog ib feem ntawm ib tug ntev kev lig kev cai ntawm kev nyob rau hauv nom tswv science rau kev siv zog los txhawb kom xaiv (Green and Gerber 2015) . Cov get-tawm-tus-pov npav thwmsim muaj ntau nyob rau hauv ib feem vim hais tias lawv muaj nyob rau hauv Pasteur lub Quadrant. Uas yog, muaj coob tus neeg uas yog mob siab rau kom pov npav thiab pov ntawv xaiv tsa yuav ua tau ib qho kev nthuav cwj pwm mus ntsuam xyuas ntau general theories txog tus cwj pwm hloov thiab kev sib raug zoo cawv.

Lwm yam kev soj ntsuam tau muab tswv yim hais txog khiav teb thwmsim nrog tus khub koom haum xws li nom tswv ob tog, NGOs, thiab cov lag luam (Loewen, Rubenson, and Wantchekon 2010; List 2011; Gueron 2002) . Lwm tus neeg muaj muaj tswv yim hais txog yuav ua li cas sib koom tes nrog cov koom haum muaj kev cuam tshuam kev tshawb fawb designs (Green, Calfano, and Aronow 2014; King et al. 2007) . Partnership yuav ua rau raug cai cov lus nug (Humphreys 2015; Nickerson and Hyde 2016) .

Tsim tswv yim (Section 4.6)

Yog hais tias koj yuav mus tsim ib qho kev soj ntsuam txoj kev npaj ua ntej yuav khiav koj xyaum ua tej yam, kuv hais tias koj pib los ntawm kev nyeem ntawv qhia raws li kev cai. Lub consort (Consolidated Standard Qhia uas raug) raws li kev cai tau tsim nyob rau hauv cov tshuaj (Schulz et al. 2010) thiab hloov rau kev sib raug zoo kev tshawb fawb (Mayo-Wilson et al. 2013) . Ib tug yam txheej txheem tau tsim los ntawm cov neeg kho ntawm lub Journal ntawm Seb Political Science (Gerber et al. 2014) (kuj saib Mutz and Pemantle (2015) thiab Gerber et al. (2015) ). Thaum kawg, qhia cov lus qhia tau tsim nyob rau hauv psychology (Group 2008) , thiab kuj saib Simmons, Nelson, and Simonsohn (2011) .

Yog hais tias koj tsim ib qho kev soj ntsuam lub tswv yim koj yuav tsum tau xav txog pre-sau npe nws vim hais tias pre-sau npe yuav ua rau kom tus cog qoob loo uas lwm tus neeg muaj nyob rau hauv koj tau. Ntxiv mus, yog hais tias koj ua hauj lwm nrog ib tug khub, nws yuav txwv koj tus khub muaj peev xwm hloov lub tsom tom qab pom cov kev tshwm sim. Pre-sau npe yog ua nce ntau nyob rau hauv psychology (Nosek and Lakens 2014) , political science (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , thiab kev lag luam (Olken 2015) .

Thaum tsim koj pre-tsom xam txoj kev npaj koj yuav tsum tau paub hais tias ib co kev soj ntsuam kuj siv regression thiab lwm yam le caag los txhim kho lub precision ntawm tus nqi kwv yees kho mob nyhuv, thiab muaj ib co sib cav tswv yim hais txog qhov no mus kom ze: Freedman (2008) , Lin (2013) , thiab Berk et al. (2013) ; saib Bloniarz et al. (2016) kom paub ntau ntxiv.

Tsim tswv yim rau hauv internet teb thwmsim yog tseem nyob rau hauv Konstan and Chen (2007) thiab Chen and Konstan (2015) .

Tsim zero nce mus nce los nqi ntaub ntawv (Section 4.6.1)

Yog xav paub ntxiv nyob rau hauv lub MusicLab thwmsim, saib Salganik, Dodds, and Watts (2006) , Salganik and Watts (2008) , Salganik and Watts (2009b) , Salganik and Watts (2009a) , thiab Salganik (2007) . Yog xav paub ntxiv nyob rau hauv khiav-coj-tag nrho kev lag luam, pom Frank and Cook (1996) . Yog xav paub ntxiv nyob rau hauv untangling koob hmoov thiab kev txawj ntau feem ntau, saib Mauboussin (2012) , Watts (2012) , thiab Frank (2016) .

Yog lwm kom ze rau xa koom nyiaj uas soj ntsuam yuav tsum tau siv nrog ceev faj: conscription. Nyob rau hauv muaj ntau yam hauv internet teb thwmsim koom yeej nais tau sau tseg rau hauv sim kawm thiab yeej tsis compensated. Piv txwv ntawm no mus kom ze xws li Restivo thiab van de Rijt tus (2012) xyaum ua tej yam rau ntshaw nyob rau hauv Wikipedia thiab daim ntawv cog lus thiab colleague tus (2012) xyaum ua tej yam rau txhawb cov neeg mus pov npav. Cov kev tsis tshua muaj pes tsawg nce mus nce los nqi, lawv muaj pes tsawg nce mus nce los them tus nqi rau kev soj ntsuam. Txawm tias tus nqi ntawm ntau yam ntawm cov kev yog tsis tshua muaj me me mus rau txhua tus neeg, me me nqi yuam ib tug enormous xov tooj ntawm cov muaj peev xwm ntxiv mus sai sai. Soj ntsuam khiav loj heev hauv internet thwmsim feem ntau nrhiav pov thawj qhov tseem ceeb ntawm me me kwv yees li ntawm kev kho mob los yog hais tias cov me me los yuav ua ib qho tseem ceeb thaum ua ntawv thov rau ntau tus neeg. Lub caij nyoog tib xav siv rau cov nqi uas soj ntsuam yaum on koom. Yog hais tias koj thwmsim ua ib lab cov neeg kom nkim ib feeb, qhov xyaum ua tej tsis yog heev teeb meem rau muaj kev tus neeg, tab sis nyob rau hauv ib pawg nws tau lub sij hawm yuav luag ob xyoos ntawm lub sij hawm.

Lwm mus kom ze rau tsim pes tsawg nce mus nce los nqi them rau koom yog siv ib tug lottery, ib mus kom ze uas tau kuj tau siv nyob rau hauv daim ntawv ntsuam xyuas kev tshawb fawb (Halpern et al. 2011) . Thaum kawg, yog xav paub ntau txog vim zoo neeg siv-yaam puab paub saib Toomim et al. (2011) .

Hloov, Cais, thiab Txo (Section 4.6.2)

Ntawm no yog cov thawj cov ntsiab lus txhais ntawm cov peb R, los ntawm Russell and Burch (1959) :

"Hloov txhais tau tias cov hloov rau meej pem nyob siab dua cov tsiaj ntawm insentient khoom. Yuav txo tau txhais tau tias yuav txo tau nyob rau hauv lub xov tooj ntawm cov tsiaj siv los muab cov ntaub ntawv ntawm ib tug muab npaum li cas thiab precision. Ca txhais tau tias tej txo nyob rau hauv cov xwm txheej los yog heev ntawm tsis tub kom cov txheej txheem siv rau cov tsiaj uas tseem muaj yuav tsum tau siv. "

Lub peb R tus uas kuv qhia tsis override qhov kev zoo hauv paus ntsiab lus piav nyob rau hauv Tshooj 6. Theej, lawv yog ib tug ntau elaborated version ib tug ntawm cov hauv paus ntsiab lus-beneficence-rau qhov chaw ntawm tib neeg thwmsim.

Thaum twg yog xav Emotional Contagion, muaj peb tsis raug cai cov teeb meem yuav tau nco ntsoov thaum txhais no xyaum ua tej yam. Ua ntej, nws tsis yog tseeb yuav ua li cas lub sij ntsiab lus ntawm qhov xyaum ua tej txuas mus rau lub theoretical neeg pab leg ntaubntawv; nyob rau hauv lwm yam lus, muaj cov lus nug txog dlaim validity. Nws tsis yog tseeb hais tias cov zoo thiab tsis zoo lo lus suav yog ua tau ib tug zoo qhia ntawm lub siab ntsws lub xeev ntawm cov neeg vim hais tias 1) nws tsis yog tseeb hais tias cov lus hais tias cov neeg rooj vag yog ib tug zoo qhia ntawm lawv tus cwj pwm txawv thiab 2) nws yog tsis meej tias cov sentiment tsom xam cov txheej txheem uas tus neeg tshawb fawb uas siv tau nti xav hais tias tus cwj pwm txawv (Beasley and Mason 2015; Panger 2016) . Nyob rau hauv lwm yam lus, muaj tej zaum yuav muaj ib tug phem tshaj plaws ntawm ib tug biased teeb liab. Ob, cov qauv thiab tsom xam ntawm qhov xyaum ua tej qhia rau peb tsis muaj dab tsi hais txog leej twg tau ntau uas raug (ie, muaj yog tsis muaj tsom xam ntawm heterogeneity ntawm kev kho mob los) li cas thiab lub mechanism tej zaum yuav. Nyob rau hauv cov ntaub ntawv no, tus neeg tshawb fawb muaj ntau ntau ntawm cov ntaub ntawv hais txog cov neeg koom, tab sis lawv yeej tseem zoo kho raws li widgets nyob rau hauv lub tsom xam. Peb, cov nyhuv loj nyob rau hauv no xyaum ua tej yam me me; qhov txawv nruab nrab ntawm tus kho mob thiab kev tswj tej yam kev mob yog hais txog 1 nyob rau hauv 1,000 lo lus. Nyob rau hauv lawv daim ntawv, Kramer thiab lug txhawb cov miv ua cov ntaub ntawv uas ib cov nyhuv ntawm no loj yog ib qho tseem ceeb vim hais tias pua pua lab ntawm cov neeg nkag tau lawv News Pub txhua hnub. Nyob rau hauv lwm yam lus, lawv cam hais tias txawm cov teebmeem uas yog me me rau txhua tus neeg uas lawv yog cov loj loj nyob hauv aggregate. Txawm hais tias koj twb los txais qhov kev sib cav, nws yog tseem tsis meej yog hais tias ib cov nyhuv ntawm no loj yog ib qho tseem ceeb hais txog qhov ntau general scientific lo lus nug txog kev xav Contagion. Yog xav paub ntxiv nyob rau hauv cov xwm txheej uas me me los yog ib qho tseem ceeb saib Prentice and Miller (1992) .

Nyob rau hauv cov nqe lus ntawm cov thawj R (Hloov), muab piv rau cov kev xav Contagion xyaum ua tej yam (Kramer, Guillory, and Hancock 2014) thiab lub siab ntsws Contagion ntuj xyaum ua tej yam (Coviello et al. 2014) muaj ib co kev qhia txog lub trade-offs muab kev koom tes nrog tsiv los ntawm thwmsim rau tej yam ntuj tso kev (thiab lwm tus le caag zoo li txuam uas los rau kwv lam thwmsim nyob rau hauv uas tsis yog-cov ntaub ntawv, saib Tshooj 2). Nyob rau hauv tas li ntawd mus rau lub koom cov kev pab cuam, switching los ntawm kev sim rau cov uas tsis-sim tshawb fawb kuj enables soj ntsuam mus kawm kev kho mob uas lawv yog ib qho uas tsis mus rau deploy. Cov raug cai thiab logistical kev pab tuaj rau ntawm ib tug nqi, Txawm li cas los. Nrog tej yam ntuj tso kev soj ntsuam ntawm muaj tsawg tswj tej yam xws li kev nrhiav neeg ntawm cov neeg koom, randomization, thiab qhov xwm ntawm cov kev kho mob. Piv txwv li, ib tug txwv ntawm dej nag raws li ib tug kho mob yog hais tias nws ob tsub kom positivity thiab txo negativity. Nyob rau hauv lub sim txoj kev tshawb no, Txawm li cas los, Kramer thiab lug txhawb cov miv twb tau mus kho positivity thiab negativity ntawm nws tus kheej.

Cov kev mus kom ze siv los ntawm Coviello et al. (2014) twb ntxiv elaborated nyob rau hauv Coviello, Fowler, and Franceschetti (2014) . Rau ib tug taw qhia rau instrumental variables saib Angrist and Pischke (2009) (tsawg formal) los yog Angrist, Imbens, and Rubin (1996) (ntau formal). Rau ib tug tsis ntseeg cais ntawm instrumental variables saib Deaton (2010) , thiab rau ib tug taw qhia rau instrumental variables nrog tsis muaj zog seev (nag yog ib tug tsis muaj zog ntsuas), saib Murray (2006) .

Feem ntau, ib tug zoo kev taw qhia rau tej yam ntuj tso thwmsim yog Dunning (2012) , thiab Rosenbaum (2002) , Rosenbaum (2009) , thiab Shadish, Cook, and Campbell (2001) muaj zoo tswv yim hais txog kwv yees causal los tsis muaj thwmsim.

Nyob rau hauv cov nqe lus ntawm lub thib ob R (kev cai), muaj scientific thiab logistical trade-offs thaum twg yog xav hloov cov qauv ntawm kev xav Contagion los ntawm thaiv cov hlab posts los boosting posts. Piv txwv li, tej zaum nws yuav hais tias cov kev siv ntawm cov xov xwm pub ua rau nws yog tshuam yooj yim ua ib tug xyaum ua tej yam nrog thaiv cov hlab posts es ib sim nrog boosting posts (nco ntsoov hais tias ib tug xyaum ua tej yam nrog thaiv cov hlab posts yuav muab los siv raws li ib tug txheej on sab saum toj ntawm cov xov xwm pub system tsis muaj kev xav tau rau alterations ntawm lwm system). Ncav, txawm li cas los, cov kev tshawb xav hais los ntwm tus xyaum ua tej yam tsis kom meej meej hais ib tsim tshaj lwm yam.

Tu siab, Kuv tsis paub txog ntau yam pauv loj ua ntej kev tshawb fawb txog tus kwv tij hauj lwm zoo ntawm thaiv cov hlab thiab boosting cov ntsiab lus nyob rau hauv cov xov xwm pub. Tsis tas li ntawd, kuv tsis tau pom ntau npaum li cas kev tshawb fawb txog kev kho mob kho kom ua rau lawv tsis muaj teeb meem; ib tug kos yog Jones and Feamster (2015) , uas suav tias yog cov ntaub ntawv ntawm kev ntsuas ntawm Internet censorship (ib lub npe kuv sib tham txog nyob rau hauv Tshooj 6 nyob rau hauv kev sib raug zoo mus rau lub Encore txoj kev tshawb no (Burnett and Feamster 2015; Narayanan and Zevenbergen 2015) ).

Nyob rau hauv cov nqe lus ntawm peb R (yuav txo tau), ib tug zoo kev taw qhia rau cov tsoos hwj chim tsom xam yog Cohen (1988) . Pre-kev kho mob covariates yuav muaj nyob rau hauv lub tsim theem thiab tus tsom xam theem ntawm kev; Tshooj 4 ntawm Gerber and Green (2012) muaj ib tug zoo kev taw qhia rau cov le caag, thiab Casella (2008) muaj ib tug ntau nyob rau hauv-tob kev kho mob. Hom kev kawm uas siv ua ntej kev kho mob cov lus qhia nyob rau hauv lub randomization yog feem ntau hu ua yog sim sim designs los yog stratified sim designs (cov terminology yog tsis siv li qhia siv kuj hla zej zog); cov hom kev kawm yog heev hais txog lub stratified zauv hom kev kawm los sib tham nyob rau hauv Tshooj 3. Saib Higgins, Sävje, and Sekhon (2016) yog xav paub ntau txog kev siv cov designs nyob rau hauv loj heev thwmsim. Pre-kev kho mob covariates kuj yuav muaj nyob rau hauv tus tsom xam theem. McKenzie (2012) txog qhov txawv-nyob rau hauv-kev txawv mus kom ze rau cais teb thwmsim nyob rau hauv ntau dua kom meej. Saib Carneiro, Lee, and Wilhelm (2016) yog xav paub ntau rau cov lag luam tawm-offs sib txawv le caag los kom precision nyob rau hauv kev kwv yees ntawm kev kho mob los. Thaum kawg, thaum txiav txim siab seb puas yuav sim kom muaj xws li pre-kev kho mob covariates ntawm tus tsim los yog tsom xam theem (los yog ob qho tib si), muaj ob peb yam tseem ceeb los xav txog. Nyob rau hauv ib qhov chaw nyob qhov twg soj ntsuam xav qhia tias lawv tsis "nuv ntses" (Humphreys, Sierra, and Windt 2013) , siv kho mob ua ntej covariates nyob rau hauv lub tsim theem yuav pab tau (Higgins, Sävje, and Sekhon 2016) . Nyob rau hauv lub sijhawm uas koom txog sequentially, tshwj xeeb tshaj yog nyob hauv internet teb thwmsim, siv pre-kev kho mob cov lus qhia nyob rau hauv lub tsim theem tej zaum yuav tsis yooj yim uas tsim, saib piv txwv li Xie and Aurisset (2016) .

Nws yog ib nqi ntxiv ib me ntsis ntawm qhov hais txog yog vim li cas txawv-nyob rau hauv-sib txawv yog li ntawd ntau npaum li cas zoo dua qhov sib txawv-nyob rau hauv-txhais tau tias. Muaj ntau online ua tau muaj siab heev variance (saib xws li, Lewis and Rao (2015) thiab Lamb et al. (2015) ) thiab no kuj ruaj khov thaum lub sij hawm. Nyob rau hauv cov ntaub ntawv no, qhov kev hloov tau yuav muaj tshuam me me variance, ua lub hwj huam ntawm lub statistical xeem. Ib tug yog vim li cas qhov no mus yog tsis siv ntau feem ntau yog tias ua ntej mus rau cov muaj hnub nyoog nws yog tsis ntau muaj pre-kev kho mob tshwm sim. Ib tug ntau pob zeb txoj kev xav txog nws yog los xav txog tej yam ib tug xyaum ua tej yam los ntsuas seb ib qho ce niaj hnub ua poob phaus. Yog hais tias koj ua ib tug txawv-nyob rau hauv-txhais tau tias mus kom ze, koj kwv yees yuav muaj variability uas los ntawm lub variability nyob rau hauv tes taw hnyav li nyob rau hauv cov pejxeem. Yog hais tias koj ua ib tug txawv-nyob rau hauv-txawv mus kom ze, txawm li cas los, hais tias lawm tshwm sim variation nyob rau hauv tes taw hnyav li twg thiaj li raug muab tshem tawm thiab koj yuav yooj yim dua ntes ib tug txawv tshwm sim los ntawm cov kev kho mob.

Ib txoj kev tseem ceeb rau txo tus naj npawb ntawm cov neeg nyob rau hauv koj xyaum ua tej yam yog kev ib lub hwj chim tsom xam, uas Kramer thiab lug txhawb cov miv yuav tau ua raws li cov nyhuv ntau thiab tsawg pab cai los ntawm lub ntuj sim los ntawm Coviello et al. (2014) los yog ua ntej lawm uas tsis yog-kev sim tshawb fawb los ntawm Kramer (2012) (nyob rau hauv qhov tseeb no yog kev ua ub no nyob rau thaum xaus ntawm tshooj no). Daim ntawv ceeb toom hais tias qhov no siv ntawm lub hwj chim tsom xam yog ib tug me ntsis txawv dua li raug. Nyob rau hauv lub analog muaj hnub nyoog, kev soj ntsuam ntawm feem ntau ua hwj huam tsom xam kom paub tseeb tias lawv cov kev tshawb no yog tsis heev me me (ie, nyob rau hauv-powered). Tam sim no, txawm li cas los, soj ntsuam yuav tsum tau ua hwj huam tsom xam kom paub tseeb tias lawv txoj kev tshawb no yog tsis heev loj (ie, tshaj-powered).

Thaum kawg, kuv pom tau hais tias muab ib tug plaub R: Repurpose. Uas yog, yog tias kev soj ntsuam ntawm pom lawv tus kheej muaj ntau cov ntaub ntawv tshaj li lawv xav nyob rau lawv tus thawj kev tshawb fawb lo lus nug, lawv yuav tsum repurpose cov ntaub ntawv nug tshiab lus nug. Piv txwv li, xav txog tej yam uas Kramer thiab lug txhawb cov miv tau siv ib tug txawv-nyob rau hauv-txawv Estimator thiab pom lawv tus kheej muaj ntau cov ntaub ntawv tshaj yuav tsum tau mus rau qhov chaw nyob cov kev tshawb fawb lo lus nug. Es tsis yog siv cov ntaub ntawv mus rau lub fullest twg, lawv yuav tau kawm qhov luaj li cas ntawm cov nyhuv raws li ib tug muaj nuj nqi rau pre-kev kho mob lub siab lub ntsws qhia. Ib yam li Schultz et al. (2007) pom tias cov nyhuv ntawm cov kev kho mob yog sib txawv rau lub teeb thiab hnyav cov neeg siv, tej zaum cov teebmeem ntawm cov xov xwm pub tau sib txawv rau cov neeg uas twb tended mus post zoo siab (los yog tu siab) lus. Repurposing yuav ua rau "nuv ntses" (Humphreys, Sierra, and Windt 2013) thiab "p-nyiag nkas" (Simmons, Nelson, and Simonsohn 2011) , tab sis cov no yog kev lom zem ntau addressable nrog ib tug ua ke ntawm kev ncaj ncees qhia (Simmons, Nelson, and Simonsohn 2011) , pre-sau npe (Humphreys, Sierra, and Windt 2013) , thiab tshuab kev kawm txoj kev uas sim kom tsis txhob tshaj-haum.