Bugu da ari, sharhin

Wannan sashe da aka tsara za a yi amfani a matsayin tunani, maimakon a karanta a matsayin labari.

• Gabatarwa (Sashe 4.1)

Tambayoyi game da causality a social bincike ne sau da yawa hadaddun da m. Ga wani foundational tsarin kula da causality bisa causal jadawalai, gani Pearl (2009) , da kuma a foundational m bisa m sakamakon, gani Imbens and Rubin (2015) (da fasaha appendix a cikin wannan sura ta). Ga wani kwatanta tsakanin wadannan biyu, ga Morgan and Winship (2014) . Ga wani m tsarin kula da ma'ana a confounder, gani VanderWeele and Shpitser (2013) .

A cikin babi na, na halitta abin da alama kamar wani haske line tsakanin mu ikon yi causal kimomi daga gwaji da kuma wadanda ba gwaji data. A gaskiya, ina tsammanin cewa fifiko ne blurrier. Alal misali, kowa da kowa ya yarda da cewa shan taba yana sa ciwon daji ko da yake ba mu taba yi a yi da ka sarrafawa gwaji cewa tilasta mutane su sha taba. Domin m littafin tsawon jiyya a kan yin causal kimomi daga wadanda ba gwaji data gani Rosenbaum (2002) , Rosenbaum (2009) , Shadish, Cook, and Campbell (2001) , da kuma Dunning (2012) .

Chapters 1 da na 2 na Freedman, Pisani, and Purves (2007) bayar da bayyana gabatarwar a cikin bambance-bambance tsakanin gwaje-gwajen, sarrafawa gwaje-gwajen, da yi da ka sarrafawa gwaje-gwajen.

Manzi (2012) na samar da wani m da zaa iya karanta gabatarwar a cikin falsafa da ilimin kididdiga underpinnings na yi da ka sarrafawa gwaje-gwajen. Har ila yau, na samar da ban sha'awa real-duniya misalai na da ikon experimentation a business.

• Mene ne gwajen? (Sashe 4.2)

Casella (2008) , Box, Hunter, and Hunter (2005) , Athey and Imbens (2016b) ta samar da mai kyau gabatarwa ga ilimin kididdiga al'amurran da gwaji zane da kuma bincike. Bugu da ari, akwai m jiyya na da amfani da gwaje-gwajen da yawa a cikin daban-daban filayen: tattalin arziki (Bardsley et al. 2009) , ilimin halayyar zaman jama'a (Willer and Walker 2007; Jackson and Cox 2013) , ilimin halin dan Adam (Aronson et al. 1989) , kimiyyar siyasa (Morton and Williams 2010) , da kuma zamantakewa da manufofin (Glennerster and Takavarasha 2013) .

Muhimmancin yar daukar ma'aikata (misali, daukan samfur) ne sau da yawa a karkashin-yaba a gwaji bincike. Duk da haka, idan sakamako na lura shi ne iri-irin da yawan jama'a, sa'an nan kuma daukan samfur ne m. Longford (1999) ya sa wannan batu a fili a lõkacin da ya umurni ga masu bincike tunanin gwaje-gwajen a matsayin yawan binciken da taragutsan daukan samfur.

• Biyu girma da gwaje-gwajen: Lab-filin da kuma analog-digital (Sashe 4.3)

The dichotomy da na gabatar a tsakanin Lab da kuma filin gwajen ne mai bit Sauki. A gaskiya, wasu masu bincike sun samarwa mafi cikakken typologies, musamman wadanda cewa raba daban-daban siffofin filin gwajen (Harrison and List 2004; Charness, Gneezy, and Kuhn 2013) . Bugu da ari, akwai biyu sauran iri-gwajen yi da zamantakewa masana kimiyya da ba su dace neatly a cikin Lab da kuma filin dichotomy: binciken gwaje-gwajen da na zamantakewa gwaje-gwajen. Survey gwaje-gwajen da ake gwaje-gwajen da yin amfani da kayayyakin da data kasance safiyo da kuma kwatanta martani ga madadin juyi na tambayoyi iri guda (wasu binciken gwaje-gwajen da aka gabatar a Babi na 3). for more on binciken gwajen ganin Mutz (2011) . Social gwajen ne gwajen inda lura shi ne wasu zamantakewa manufofin da za a iya aiwatar da gwamnati. Social gwaje-gwajen da kakkarfan alaka shirin tantancewa. Don ƙarin a kan manufofin gwaje-gwajen, gani Orr (1998) , Glennerster and Takavarasha (2013) , da kuma Heckman and Smith (1995) .

A yawan takardun sun kwatanta Lab da kuma filin gwaje-gwajen a cikin m (Falk and Heckman 2009; Cialdini 2009) da kuma cikin sharuddan sakamakon takamaiman gwaje-gwajen a fannin kimiyyar siyasa (Coppock and Green 2015) , da tattalin arziki (Levitt and List 2007a; Levitt and List 2007b; Camerer 2011; Al-Ubaydli and List 2013) da kuma ilimin halin dan Adam (Mitchell 2012) . Jerit, Barabas, and Clifford (2013) yayi wani m bincike zane ga gwada sakamakon daga Lab da kuma filin gwaje-gwajen.

Damuwa game da mahalarta canza musu hali, domin sun san ana hankali lura wani lokaci ana kiran bukatar effects, kuma suka kasance sunã yi karatu a tunani (Orne 1962) da kuma tattalin arziki (Zizzo 2009) . Ko da yake mafi yawa hade da Lab gwaje-gwajen, wadannan guda al'amurran da suka shafi zai iya sa matsaloli ga filin gwaje-gwajen da. A gaskiya ma, bukatar effects ma wani lokaci da ake kira Hawthorne effects, a wani ajali wanda ya sami asali daga wata filin gwaji, musamman sanannen haske gwaje-gwajen da ya fara a 1924 a Hawthorne Works na Western Electric Company (Adair 1984; Levitt and List 2011) . Dukansu bukatar effects da Hawthorn effects suna a hankali alaka da ra'ayin amsawa ji tattauna a Babi na 2 (duba Webb et al. (1966) ).

A tarihin filin gwaje-gwajen da aka bayyana a cikin tattalin arziki (Levitt and List 2009) , kimiyyar siyasa (Green and Gerber 2003; Druckman et al. 2006; Druckman and Lupia 2012) , ilimin halin dan Adam (Shadish 2002) , da kuma jama'a da manufofin (Shadish and Cook 2009) . Daya fannin ilmin zaman inda filin gwaje-gwajen da sauri ya zama shahararren ne na kasa da kasa ci gaba. Ga wani m review na cewa aiki a cikin tattalin arziki ga Banerjee and Duflo (2009) , da kuma wani m kima ga Deaton (2010) . Ga wani review wannan aikin a fannin kimiyyar siyasa ga Humphreys and Weinstein (2009) . A karshe, cikin da'a kalubale da hannu tare da filin gwaje-gwajen da aka bincika a fannin kimiyyar siyasa (Humphreys 2015; Desposato 2016b) da kuma ci gaban tattalin arziki (Baele 2013) .

A cikin babi na, na nuna cewa pre-jiyya bayani za a iya amfani da su inganta sahihancin kiyasta magani effects, amma akwai wasu muhawara game da wannan dabarar: Freedman (2008) , Lin (2013) , da kuma Berk et al. (2013) . ganin Bloniarz et al. (2016) don ƙarin bayani.

• Motsi bayan sauki gwajen (Sashe 4.4)

Na zaba da hankali a kan uku Concepts: tushe, heterogeneity magani effects, kuma sunadaran. Wadannan Concepts da sunaye daban-daban a bangarori daban-daban. Alal misali, masana ilimin tunani na ayan motsa bayan sauki gwaje-gwajen da mayar da hankali a kan shiga tsakani da sulhu (Baron and Kenny 1986) . A ra'ayin na shiga tsakani da aka kama da abin da na kira sunadaran, da kuma ra'ayin sulhu ne kama da abin da na kira external tushe (misali, zai sakamakon gwajin zama daban-daban, idan aka gudanar a yanayi daban-daban), kuma heterogeneity magani effects ( misali, su ne effects ya fi girma ga wasu mutane fiye da sauran mutane).

The gwajin na Schultz et al. (2007) ya nuna yadda zamantakewa theories za a iya amfani da su tsara tasiri shisshigi. Domin a more general shawara game da rawar da ka'idar a zayyana tasiri shisshigi, gani Walton (2014) .

• Tushe (Sashe 4.4.1)

The Concepts na ciki da na waje tushe An fara gabatar a Campbell (1957) . Dubi Shadish, Cook, and Campbell (2001) domin karin cikakken tarihi da kuma mai da hankali elaboration na ilimin kididdiga ƙarshe tushe, ciki tushe, yi inganci, kuma external tushe.

Domin wani bayyani na al'amurran da suka shafi alaka ilimin kididdiga ƙarshe tushe a gwaje-gwajen da ganin Gerber and Green (2012) (domin a ilmin zaman hangen zaman gaba) da kuma Imbens and Rubin (2015) (domin a ilimin kididdiga hangen zaman gaba). Wasu al'amurran da suka shafi na ilimin kididdiga ƙarshe tushe cewa taso musamman a online filin gwaje-gwajen sun hada da batutuwa kamar computationally m hanyoyin samar da amincewa jinkiri da dogara data (Bakshy and Eckles 2013) .

Ciki tushe iya zama da wahala don tabbatar a hadaddun filin gwaje-gwajen. Duba, ga misali, Gerber and Green (2000) , Imai (2005) , da kuma Gerber and Green (2005) don muhawara game da aiwatar da wani hadadden filin gwajin game zabe. Kohavi et al. (2012) da kuma Kohavi et al. (2013) ta samar da wani gabatarwar a cikin kalubalen tazara tushe a online filin gwaje-gwajen.

Daya babbar damuwa da ciki tushe ne matsalolin da randomization. Daya hanyar yiwuwar gane matsalolin da randomization shine gwada lura da kuma kula da kungiyoyin a Fitowan halaye. Wannan irin kwatanta da aka kira wani balance rajistan shiga. Dubi Hansen and Bowers (2008) a ilimin kididdiga m daidaita cak, da kuma ganin Mutz and Pemantle (2015) domin damuwa game balance cak. Alal misali, ta amfani da balance duba Allcott (2011) gano cewa akwai wasu shaida cewa randomization ba aiwatar daidai a uku daga cikin gwaje-gwajen a wasu gwaje-gwajen da OPower (duba Table 2; sites 2, 6, da 8). Don wasu hanyoyin, gani Imbens and Rubin (2015) , Babi na 21.

Sauran manyan damuwa alaka ciki tushe su ne: 1) daya gefe ba sharadi, inda ba kowa da kowa a lura kungiyar zahiri karbi magani, 2) biyu gefe ba sharadi, inda ba kowa da kowa a lura kungiyar na'am da magani da kuma wasu mutane a cikin iko kungiyar sama da magani, 3) attrition, inda sakamakon da aka ba auna ga wasu mahalarta, da kuma 4) tsangwama, inda magani spills kan daga mutãne, a cikin magani yanayin da mutane a cikin iko yanayin. Dubi Gerber and Green (2012) Chapters 5, 6, 7, 8 da kuma don ƙarin kan kowane daga cikin wadannan al'amurra.

Domin more on gina tushe, gani Westen and Rosenthal (2003) , da kuma more on gina tushe a babban data kafofin, Lazer (2015) da kuma Babi na 2 na wannan littafin.

Daya bangare na external tushe ne saitin inda wani baki da aka gwada. Allcott (2015) na samar da wani m msar tambayar da empirical lura da shafin selection nuna bambanci. Wannan fitowar kuma tattauna a Deaton (2010) . Bugu da kari da ake replicated da yawa a cikin sites, Home Energy Report baki kuma an da kansa binciken da mahara bincike kungiyoyin (misali, Ayres, Raseman, and Shih (2013) ).

• Heterogeneity magani effects (Sashe 4.4.2)

Domin mai kyau bayyani na heterogeneity magani effects a filin gwaje-gwajen, ka duba babi na 12 na Gerber and Green (2012) . Domin gabatarwa zuwa heterogeneity magani effects a likita gwaji, gani Kent and Hayward (2007) , Longford (1999) , da kuma Kravitz, Duan, and Braslow (2004) . Heterogeneity magani effects kullum mayar da hankali a kan bambance-bambance bisa pre-jiyya halaye. Idan kun kasance interested in heterogeneity bisa post-magani sakamakon, to more hadaddun approachs ake bukata kamar babba stratification (Frangakis and Rubin 2002) . ga Page et al. (2015) a review.

Mutane da yawa masu bincike kimanta da heterogeneity magani effects amfani mikakke komawa da baya, amma sababbin hanyoyin dogara na'ura koyo, misali Green and Kern (2012) , Imai and Ratkovic (2013) , Taddy et al. (2016) , da kuma Athey and Imbens (2016a) .

Akwai wasu shakka game da binciken na heterogeneity na effects saboda mahara kwatanta matsaloli da kuma "fishing." Akwai da dama na ilimin kididdiga ta kusance da za su iya taimaka address damuwa game da mahara kwatanta (Fink, McConnell, and Vollmer 2014; List, Shaikh, and Xu 2016) . Daya tsarin kula da damuwa game da "fishing" shi ne pre-rajista, wanda aka ƙara zama kowa a tunani (Nosek and Lakens 2014) , kimiyyar siyasa (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , da kuma tattalin arziki (Olken 2015) .

A cikin nazarin Costa and Kahn (2013) ne kawai game da rabi daga cikin gidaje a cikin gwaji su iya a nasaba da alƙaluma da bayanai. Masu karatu sha'awar da cikakken bayani kuma zai yiwu matsaloli tare da wannan bincike ya kamata koma zuwa ainihin takarda.

• Sassan (Sashe 4.4.3)

Sassan ne wuce yarda da muhimmanci, amma suka juya a kira su da wuya a nazarin. Research game sunadaran hankali alaka da nazarin shiga tsakani a tunani (amma ga kuma VanderWeele (2009) a daidai kwatanta tsakanin biyu ideas). Ilimin kididdiga hanyoyin gano hanyoyin, kamar m ci gaba a Baron and Kenny (1986) , su ne quite na kowa. Abin baƙin ciki, shi dai itace cewa wadanda hanyoyin dogara ne a kan wasu m zaton (Bullock, Green, and Ha 2010) da kuma sha a lokacin da akwai mahara sunadaran, a matsayin daya iya sa ran da yawa a cikin yanayi (Imai and Yamamoto 2013; VanderWeele and Vansteelandt 2014) . Imai et al. (2011) da kuma Imai and Yamamoto (2013) bayar da wasu ingantattun ilimin kididdiga hanyoyi. Bugu da ari, VanderWeele (2015) yayi wani littafi-tsawon lura da dama da muhimmanci results, ciki har da wani m tsarin kula da ji na ƙwarai analysis.

A raba m mayar da hankali a kan gwaje-gwajen da cewa ƙoƙari ya yi amfani da inji kai tsaye (misali, bada sailors bitamin C). Abin baƙin ciki, mutane da yawa a cikin ilmin zaman saituna akwai sau da yawa mahara sunadaran da yana da wuya a tsara jiyya da canja daya ba tare da canza wasu. Wasu hanyoyin gwaje musanyãwa sunadaran da aka bayyana a cikin Imai, Tingley, and Yamamoto (2013) , Ludwig, Kling, and Mullainathan (2011) , da kuma Pirlott and MacKinnon (2016) .

A karshe, sunadaran ma da dogon tarihi a falsafar kimiyya kamar yadda aka bayyana da Hedström and Ylikoski (2010) .

• Amfani data kasance muhallin (Sashe 4.5.1.1)

Don ƙarin a kan yin amfani da rubutu da karatu da kuma duba karatu don auna da nuna bambanci ga Pager (2007) .

• Gina your own gwaji (Sashe 4.5.1.2)

Mafi na kowa hanyar kurtu mahalarta su gwaje-gwajen da za ka gina shi ne Amazon Mechanical Turk (MTurk). Saboda MTurk mimics al'amurran gargajiya Lab gwaje-gwajen-biya mutane su kammala ayyuka da cewa su ba zai yi ba for free-da yawa masu bincike sun riga ya fara yin amfani da Turkers (da ma'aikata a kan MTurk) kamar yadda mahalarta a mutum batutuwa gwajen sakamakon a cikin sauri kuma mai rahusa data tarin fiye da na gargajiya on-harabar dakin gwaje-gwaje gwaje-gwajen (Paolacci, Chandler, and Ipeirotis 2010; Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012; Rand 2012; Berinsky, Huber, and Lenz 2012) .

Babbar ƙarfin gwaje-gwajen da mahalarta dauka daga MTurk ne tafarkin: sun ba da damar bincike kurtu mahalarta da sauri, kuma kamar yadda ake bukata. Ganin cewa Lab gwaje-gwajen iya daukar makonni gudu da kuma filin gwajen iya daukar watanni domin saita-up, gwaje-gwajen da mahalarta dauka daga MTurk za a iya gudu a days. Alal misali, Berinsky, Huber, and Lenz (2012) sun iya kurtu 400 batutuwa guda daga yini shiga a cikin wani 8 minti gwaji. Bugu da ari, wadannan mahalarta za a iya dauka domin kusan duk wani dalili (ciki har da safiyo da kuma taro da haɗin gwiwar, kamar yadda aka tattauna a Babi na 3 da 5). Wannan sauƙi na daukar ma'aikata yana nufin cewa masu bincike za su iya gudanar da jerin daga related gwaje-gwajen a m maye.

Kafin a jawo ra'ayinsu mahalarta daga MTurk for your own gwaje-gwajen, akwai huɗu da muhimmanci abubuwa su sani. Na farko, da yawa masu bincike da wadanda ba musamman shakka daga gwaje-gwajen shafe Turkers. Domin wannan shakka ba musamman, yana da wuya a} alubalantar da shaida. Duk da haka, bayan shekaru da dama na karatu ta yin amfani da Turkers, yanzu muna iya cewa wannan shakka ba musamman zama dole. A nan an yawa karatu gwama demographics na Turkers zuwa wasu alƙarya da yawa karatu gwada sakamakon gwaje-gwajen da Turkers da sakamakon daga wasu jama'a. Ganin dukan wannan aiki, na yi tunanin cewa hanya mafi kyau a gare ka ka yi tunani game da shi shi ne cewa Turkers ne m saukaka samfurin, da yawa kamar dalibai amma dan kadan more bambancin (Berinsky, Huber, and Lenz 2012) . Saboda haka, kamar yadda dalibai ne a m yawan ga wasu amma ba dukan gwaji bincike, Turkers ne m yawan ga wasu amma ba dukan bincike. Idan za ku yi aiki tare da Turkers, to, shi ya sa hankali ya karanta da dama daga cikin wadannan kamanta karatu da kuma fahimtar nuances.

Na biyu, masu bincike sun ɓullo da m-ayyuka na kara ciki inganci na Turk gwaje-gwajen, da ya kamata ka koyi game da bi wadannan m-ayyuka (Horton, Rand, and Zeckhauser 2011; Mason and Suri 2012) . Alal misali, masu bincike, ta yin amfani da Turkers suna karfafa yin amfani da screeners cire gafalallu mahalarta (Berinsky, Margolis, and Sances 2014; Berinsky, Margolis, and Sances 2016) (amma ga kuma DJ Hauser and Schwarz (2015b) da kuma DJ Hauser and Schwarz (2015a) ). Idan ba ka cire gafalallu mahalarta, to, duk wani lahanin da za a iya lura wanke fitar da amo gabatar daga gafalallu mahalarta, kuma a yi yawan gafalallu mahalarta iya zama gwaji. A cikin gwajin da Huber da kuma abokan aiki (2012) game da 30% na mahalarta kasa na asali hankali screeners. Wani al'amari na kowa da Turkers ne wadanda ba butulci mahalarta (Chandler et al. 2015) .

Na uku, dangi zuwa wasu siffofin digital gwaje-gwajen, MTurk gwaje-gwajen ba zai iya hawansa. Stewart et al. (2015) ya yi kiyasin cewa, a kowace lokaci akwai kawai game da 7,000 mutane a kan MTurk.

A karshe, ya kamata ka san cewa MTurk akwai wata al'umma tare da kansa dokoki da kuma norms (Mason and Suri 2012) . A wannan hanyar da za ka yi kokarin gano game da al'adun a kasar inda kuka kasance faruwa gudu your gwaje-gwajen, ya kamata ka yi kokarin gano game da al'adu da kuma norms na Turkers (Salehi et al. 2015) . Kuma, ya kamata ka san cewa Turkers za a magana game da gwaji idan ka yi wani abu da bai dace ba ko unethical (Gray et al. 2016) .

MTurk ne mai wuce yarda m hanyar kurtu mahalarta to your gwaje-gwajen, ko suna Lab-kamar, kamar Huber, Hill, and Lenz (2012) , ko fiye da filin-kamar, kamar Mason and Watts (2009) , Goldstein, McAfee, and Suri (2013) , Goldstein et al. (2014) , Horton and Zeckhauser (2016) , da kuma Mao et al. (2016) .

• Gina your own samfur (Sashe 4.5.1.3)

Idan kana tunanin kokarin haifar your own samfur, ina bada shawara cewa ka karanta shawara miƙa ta MovieLens kungiyar a Harper and Konstan (2015) . A key m daga kwarewa ne cewa ga kowane nasara shiri akwai mutane da yawa, mutane da yawa kasawa. Alal misali, MovieLens kungiyar kaddamar sauran kayayyakin, irin su GopherAnswers da suke da cikakken kasawa (Harper and Konstan 2015) . Wani misali da wani bincike yayin da kasawa yunkurin kafa wani samfurin ne Edward Castronova ta yunkurin gina wani online game kira Arden. Duk da $ 250,000 a kudade, aikin ya Flop (Baker 2008) . Projects kamar GopherAnswers da Arden ne da rashin alheri fiye da kowa fiye da ayyukan kamar MovieLens. A karshe, a lokacin da na ce zan ba su sani ba daga wani bincike da suka samu nasarar gina kayayyakin maimaita experimentation a nan su ne ta sharudda: 1) mahalarta amfani da samfur saboda abin da ya ciyar da su (misali, ba su da biya kuma ba su da masu sa kai taimaka kimiyya) da kuma 2) samfurin da aka yi amfani da shi fiye da daya jinsin gwaji (ie, ba iri daya ba gwaji mahara sau da daban-daban yar wuraren waha). Idan ka san wasu misalai, don Allah a sanar da ni.

• Partner da iko (Sashe 4.5.2)

Na kuma sha jin ra'ayin Pasteur ta Quadrant tattauna akai-akai a tech kamfanonin, da kuma yana taimaka shirya bincike} o} arin a Google (Spector, Norvig, and Petrov 2012) .

Bond da kuma abokan aiki 'binciken (2012) kuma ƙoƙarin gane da sakamako daga cikin wadannan jiyya a kan abokai wadanda suka karbi su. Domin daga cikin zane na gwaji, wadannan spillovers ne wuya a gane cleanly. sha'awar karatu ya kamata ga Bond et al. (2012) domin karin sosai tattaunawa. Wannan gwaji ne na dogon al'adar gwaje-gwajen a fannin kimiyyar siyasa a kokarin karfafa 'yancin kada kuri'a (Green and Gerber 2015) . Wadannan samu-fito-da-zaben gwajen ne na kowa a sashi domin sun kasance a cikin Pasteur ta Quadrant. Wancan ne, akwai mutane da yawa da suke motsa ka ka ƙara zabe kuma zabe na iya zama mai ban sha'awa hali ya gwada more general theories game hali canji da kuma zamantakewa tasiri.

Other masu bincike sun bayar da shawara game da guje filin gwaje-gwajen da abokin tarayya kungiyoyi kamar jam'iyyun siyasa, kungiyoyi masu zaman kansu, da kuma harkokin kasuwanci (Loewen, Rubenson, and Wantchekon 2010; List 2011; Gueron 2002) . Wasu sun miƙa shawara game da yadda tarayya da kungiyoyin iya tasiri bincike kayayyaki (Green, Calfano, and Aronow 2014; King et al. 2007) . Partnership kuma iya haifar da da'a tambayoyi (Humphreys 2015; Nickerson and Hyde 2016) .

• Design shawara (Sashe 4.6)

Idan za ka ƙirƙiri wani bincike da shirin da a guje your gwaji, na bayar da shawarar cewa ka fara da karanta rahoto jagororin. The jima'i (Consolidated Standard Rahoto na gwaji) jagororin aka ɓullo da a magani (Schulz et al. 2010) da kuma modified ga zamantakewa bincike (Mayo-Wilson et al. 2013) . A related sa na jagororin an ci gaba da gyara da Journal of gwajin Kimiyyar Siyasa (Gerber et al. 2014) (duba Mutz and Pemantle (2015) da kuma Gerber et al. (2015) ). A karshe, rahoton jagororin da aka ɓullo da a tunani (Group 2008) , da kuma duba Simmons, Nelson, and Simonsohn (2011) .

Idan ka ƙirƙiri wani bincike da shirin ya kamata ka yi la'akari da pre-rijista shi domin pre-rajista zai kara amincewa da wasu da ku a cikin results. Bugu da ari, idan kana da aiki tare da wani abokin tarayya, zai rage your abokin tarayya ta ikon canja analysis bayan ganin sakamakon. Pre-rajista an ƙara zama kowa a tunani (Nosek and Lakens 2014) , kimiyyar siyasa (Humphreys, Sierra, and Windt 2013; Monogan 2013; Anderson 2013; Gelman 2013; Laitin 2013) , da kuma tattalin arziki (Olken 2015) .

Duk da yake samar da your pre-bincike shirin ya kamata ka sani cewa wasu masu bincike kuma amfani da komawa da baya da kuma related hanyoyin inganta daidaici na kiyasta magani sakamako ne, kuma babu wani muhawara game da wannan dabarar: Freedman (2008) , Lin (2013) , da kuma Berk et al. (2013) . ganin Bloniarz et al. (2016) don ƙarin bayani.

Design shawara musamman domin online filin gwaje-gwajen da aka kuma gabatar a cikin Konstan and Chen (2007) da kuma Chen and Konstan (2015) .

• Create sifili m kudin data (Sashe 4.6.1)

Don ƙarin a kan MusicLab gwaje-gwajen, gani Salganik, Dodds, and Watts (2006) , Salganik and Watts (2008) , Salganik and Watts (2009b) , Salganik and Watts (2009a) , kuma Salganik (2007) . Don ƙarin on lashe-sha-duka kasuwanni, gani Frank and Cook (1996) . Don ƙarin on untangling luck da fasaha more kullum, gani Mauboussin (2012) , Watts (2012) , da kuma Frank (2016) .

Akwai wani tsarin kula da kawar yar biya cewa masu bincike ya kamata amfani da hankali: conscription. A da yawa online filin gwajen mahalarta suna m tsara a cikin gwaje-gwajen, kuma bai taba cika. Misalan wannan dabarar hada Restivo kuma van de Rijt ta (2012) gwaji a kan lada a Wikipedia da Bond da kuma abokin aiki ta (2012) gwaji a kan ƙarfafa mutane su kada kuri'a. Wadannan gwaje-gwajen ba gaske da sifilin m cost, suna da sifilin m kudin bincike. Ko da yake kudin da yawa daga cikin wadannan gwaje-gwajen ne musamman kananan zuwa kowane ɗan takara, kananan halin kaka hõre wani babban yawan mahalarta iya ƙara sama sauri. Masu bincike a guje m online gwajen sau da yawa tabbatar da muhimmancin kananan kiyasta magani effects da cewa wadannan kananan effects iya zama da muhimmanci a lõkacin da amfani ga mutane da yawa. A daidai wannan tunani ya shafi halin kaka cewa masu bincike kallafã mahalarta. Idan gwajen sa miliyan daya mutane su vata minti daya, da gwajin ne ba sosai cutarwa ga wani musamman mutum, amma a tara da ya kuka da kansa kusan shekaru biyu lokaci.

Wani tsarin kula da samar da sifili m kudin biya zuwa mahalarta ne a yi amfani da irin caca, an m cewa ya kuma an yi amfani da binciken da bincike (Halpern et al. 2011) . A karshe, ga mafi game da zayyana m amfani-abubuwan gani Toomim et al. (2011) .

• Sauya, sake tace, kuma Rage (Sashe 4.6.2)

A nan ne na farko ma'anar uku R, daga Russell and Burch (1959) :

"Sauyawa nufin canzawa ga m rayuwa mafi girma dabbobi da insentient abu. Saukarwa nufin rage a cikin lambobin dabbobi amfani da su samu bayanai daga wani da aka ba adadin da daidaici. Tsaftacewa nufin wani karu a faru ko tsananin tausayi hanyoyin amfani ga waɗanda dabbobi wanda har yanzu da za a amfani da su. "

The uku R ta cewa zan ba da shawara ba override da da'a ka'idojin da aka bayyana a Babi na 6. Ã'a, sũ more elaborated version daya daga cikin wadanda ka'idojin-karimci-musamman domin ba da lõkutan fãɗuwar mutum gwaje-gwajen.

A lokacin da la'akari Wani tunanin Contagion, akwai uku wadanda ba da'a al'amurran da suka shafi ci gaba da tuna a lõkacin da tafsirin wannan gwaji. Na farko, shi ne, ba bayyana yadda ainihin cikakken bayani game da gwajin gama da msar tambayar da'awar. a cikin wasu kalmomi, akwai tambayoyi game da gina tushe. Da ba shi da bayyana a fili cewa tabbatacce kuma korau kalmar kirga su ne ainihin mai kyau nuna alama daga cikin tunanin Jihar mahalarta saboda 1) shi ne, ba a fili cewa maganar da mutane post ne mai kyau nuna alama na da motsin zuciyarmu da kuma 2) shi ne, ba a fili cewa musamman jin zuciya analysis m cewa masu bincike amfani da shi ne iya dogara infer motsin zuciyarmu (Beasley and Mason 2015; Panger 2016) . A wasu kalmomin, akwai zai yi wani mummunan gwargwado na son zuciya sigina. Na biyu, da zane da kuma bincike na gwaji ya gaya mana kome ba game da wanda aka fi tasiri (ie, babu wani bincike na heterogeneity magani effects), da abin da inji zai yi. A wannan yanayin, da masu bincike da kuri'a na bayanai game da mahalarta, amma sun kasance da gaske bi da matsayin Widgets a analysis. Na uku, da sakamako size a cikin wannan gwajin ne ƙwarai kananan. bambanci tsakanin jiyya da kuma kula da yanayin ne game da 1 a 1,000 kalmomi. A cikin takarda, Kramer da kuma abokan aiki yi haka al'amarin da cewa wani sakamako na wannan size yana da muhimmanci domin daruruwan miliyoyin mutane samun damar News Feed kowace rana. A wasu kalmomin, suka yi jayayya cewa, ko da effects da cewa su ne kananan ga kowane mutum su ne babban a tara. Ko da kun kasance a yarda da wannan shawara, shi ne har yanzu ba a share idan wani sakamako na wannan size yana da muhimmanci game da more general kimiyya tambaya game da wani tunanin contagion. Don ƙarin a kan yanayi inda kananan effects suna da muhimmanci ga Prentice and Miller (1992) .

A cikin sharuddan na farko R (Sauyawa), gwada wani tunanin Contagion gwaji (Kramer, Guillory, and Hancock 2014) da kuma wani tunanin contagion halitta gwaji (Coviello et al. 2014) yayi wasu general darussa game da cinikayya-offs hannu da motsi daga gwaje-gwajen a gwaje-gwajen da na halitta (da kuma sauran hanyoyin kamar matching cewa ƙoƙari m gwaje-gwajen a ba-gwaji data, ka duba babi na 2). Baya ga da'a alfarma, ya sauya sheka daga gwaji ga wadanda ba gwaji karatu ma sa masu bincike ya yi nazarin jiyya cewa suna logistically iya tura. Wadannan da'a da kuma tafarkin amfanin zo a cost, duk da haka. Tare da halitta gwaje-gwajen da masu bincike da kasa da iko a kan abubuwa kamar daukar ma'aikata na mahalarta, randomization, da kuma yanayin da magani. Alal misali, daya ya rage mata ruwan sama a matsayin magani shi ne cewa shi biyu qara positivity da rage-rage negativity. A cikin gwajin binciken, duk da haka, Kramer da kuma abokan aiki sun iya daidaita positivity da negativity da kansa.

The musamman m amfani da Coviello et al. (2014) ya kara bada haske a Coviello, Fowler, and Franceschetti (2014) . Domin an gabatarwar instrumental canji ga Angrist and Pischke (2009) (m m) ko Angrist, Imbens, and Rubin (1996) (more m). Ga wani m kima na instrumental canji ga Deaton (2010) , da kuma wani gabatarwar instrumental canji da rauni kida (ruwan sama ne mai rauni kayan aiki), gani Murray (2006) .

More kullum, mai kyau gabatarwar halitta gwajen ne Dunning (2012) , da kuma Rosenbaum (2002) , Rosenbaum (2009) , da kuma Shadish, Cook, and Campbell (2001) bayar da kyau ra'ayi game da kimantawa causal effects ba tare da gwaje-gwajen.

A cikin sharuddan na biyu R (tsaftacewa), akwai kimiyya da kuma tafarkin cinikayya-offs lokacin da la'akari canza zane na Wani tunanin Contagion daga tarewa posts to boosting posts. Alal misali, yana iya zama haka al'amarin da cewa fasaha aiwatar da News Feed sa shi ne ma fi sauƙi ga aikata wani gwaji tare da tarewa posts maimakon wani gwaji tare da gonakin posts (a lura da cewa an gwaji tare da tarewa posts za a iya aiwatar da matsayin Layer a kan saman News Feed tsarin ba tare da wani bukatar gyare-gyare da tamkar tsarin). Kimiyance, duk da haka, ka'idar jawabi da gwaji ba a fili bayar da shawarar daya zane a kan sauran.

Abin baƙin ciki, ba ni sane da gwaji kafin bincike game da dangi isa yabo na tarewa da gonakin abun ciki a cikin News Feed. Har ila yau, na ba su gani ba da yawa bincike game refining jiyya su sa su kasa cutarwa. daya banda shi ne Jones and Feamster (2015) , wanda ya ɗauki akwati na ji na Internet katsalandan (a topic na tattauna a Babi na 6 a dangantaka da Encore binciken (Burnett and Feamster 2015; Narayanan and Zevenbergen 2015) ).

A cikin sharuddan na uku R (raguwa), mai kyau gabatarwar gargajiya ikon analysis ne Cohen (1988) . Pre-jiyya covariates za a iya kunshe a cikin zane mataki da bincike mataki na gwaje-gwajen. Babi na 4 na Gerber and Green (2012) na samar da mai kyau gabatarwar biyu fuskanci, kuma Casella (2008) na samar da wani more in-zurfin magani. Dabarun da suke amfani da wannan pre-jiyya bayani a cikin randomization suna yawanci ake kira ko dai katange gwaji kayayyaki ko rabe gwaji kayayyaki (da terminology ba a amfani da consistently fadin al'umma). wadannan fasahohi suna warai alaka da rabe daukan samfur dabaru tattauna a Babi na 3. See Higgins, Sävje, and Sekhon (2016) domin more on yin amfani da wadannan kayayyaki a cikin m gwaje-gwajen. Pre-jiyya covariates kuma za a iya kunshe a cikin analysis mataki. McKenzie (2012) duba bambanci-in-bambance-bambance tsarin kula da nazarin filin gwaje-gwajen a zurfafe. Dubi Carneiro, Lee, and Wilhelm (2016) don ƙarin kan cinikayya-offs tsakanin daban-daban fuskanci ƙara daidaici a kimomi na lura effects. A karshe, lokacin da yankan shawara ko don kokarin hada pre-jiyya covariates a zane ko analysis mataki (ko biyu), akwai 'yan dalilai yi la'akari. A cikin wata saitin inda masu bincike ke so ya nuna cewa su ba "fishing" (Humphreys, Sierra, and Windt 2013) , ta yin amfani da pre-jiyya covariates a cikin zane mataki zai iya zama taimako (Higgins, Sävje, and Sekhon 2016) . A yanayi inda mahalarta zo sequentially, musamman online filin gwaje-gwajen, ta yin amfani da pre-jiyya bayanai a cikin zane mataki na iya zama da wahala logistically, gani misali Xie and Aurisset (2016) .

Yana da daraja daša bit na diraya game dalilin da ya sa bambanci-in-bambance-bambance zai iya zama haka yafi tasiri fiye da bambanci-in-wajen. Mutane da yawa online sakamakon da sosai high sãɓã wa jũna. (Duba misali, Lewis and Rao (2015) da kuma Lamb et al. (2015) ) kuma suna gwada barga a kan lokaci. A wannan yanayin, da canjin ci zai yi ma karami sãɓã wa jũna, kara ikon ilimin kididdiga gwajin. Wani dalili da wannan kusata ba a amfani da mafi sau da yawa ne cewa kafin a digital shekaru shi ba kowa a yi pre-jiyya sakamakon. A more kankare hanyar tunani game da shi shi ne su yi tunanin wani gwaji don auna ko wani darasi na yau da kullum yana sa nauyi asara. Idan ka yi wani bambanci-in-wajen m, your kimanta su da canzawa da cewa ya zo daga canzawa a nauyi a cikin yawan. Idan ka yi wani bambanci-in-bambanci m, duk da haka, cewa halitta abin da ke faruwa bambancin a nauyi samun kawar da za ka iya more sauƙi gane bambanci sa da magani.

Ɗaya daga cikin muhimman hanya don rage yawan mahalarta a cikin gwaji ne da za su gudanar da wani iko analysis, wanda Kramer da kuma abokan aiki zai iya yi dangane da sakamako masu girma dabam lura daga halitta gwaji da Coviello et al. (2014) ko a baya ba gwaji bincike da Kramer (2012) (a gaskiya wadannan su ne ayyukan a karshen wannan babi). Ka lura da cewa wannan amfani da ikon bincike ne a bit daban-daban fiye da hankula. A cikin analog shekaru, masu bincike kullum yi iko analysis don tabbatar da cewa su yi karatu ba ma kananan (ie, a karkashin-Powered). Yanzu, duk da haka, masu bincike ya yi ƙarfi analysis don tabbatar da cewa su yi karatu ba ma babban (ie, a kan-Powered).

A karshe, ina dauke da ƙara a karo na hudu R: Repurpose. Wato, idan masu bincike gano kansu da more gwaji data daga gare su bukatar magance su na asali bincike tambaya, ya kamata su repurpose da bayanai zuwa tambayar sabon tambayoyi. Alal misali, tunanin cewa Kramer da kuma abokan aiki ya yi amfani da wani bambanci-in-bambance-bambance estimator kuma sami kansu da more data fiye da ake bukata don magance su bincike tambaya. Maimakon ba ta yin amfani da bayanai zuwa cikakkiyar har, za su iya yi binciken da girman da sakamako a matsayin aiki to pre-jiyya tunanin magana. Kamar yadda Schultz et al. (2007) ya gano cewa sakamakon da jiyya ya daban-daban ga haske da kuma nauyi users, watakila alãmõmin News Feed kasance daban-daban ga mutanen da suka riga kula to post m (ko m) saƙonni. Repurposing zai iya haifar da "fishing" (Humphreys, Sierra, and Windt 2013) da kuma "p-shiga ba tare da izini ba" (Simmons, Nelson, and Simonsohn 2011) , amma wadannan su ne sun fi mayar addressable da hade da gaskiya rahoto (Simmons, Nelson, and Simonsohn 2011) , pre-rajista (Humphreys, Sierra, and Windt 2013) , da kuma na'ura koyo hanyoyin da ƙoƙari don kauce wa a kan-kasancẽwa.