Activities

Key:

  • dhigirii yakaoma: nyore nyore , svikiro nzira , nesimba zvakaoma , Zvakaoma zvakaoma
  • kunoda masvomhu ( kunoda masvomhu )
  • kunoda Coding ( kunoda Coding )
  • date muunganidzwa ( date muunganidzwa )
  • vaaida vangu ( wangu kufarira )

  1. [ nzira , wangu kufarira ] Algorithmic akakunda aiva nechinetso Google flu Trends. Verenga pepa rakanyorwa Lazer et al. (2014) , uye kunyora pfupi, yakajeka tsambambozha kuti injiniya panguva Google vachitsanangura dambudziko uye kupa pfungwa sei kugadzirisa dambudziko.

  2. [ nzira ] Bollen, Mao, and Zeng (2011) anoti kuti mashoko kubva Twitter zvingashandiswa kufanotaura pamusika wemari. Kuwana uku kwakaita kuti paumbwe noruzhowa mari-Derwent Capital Markets-mari yake pamusika wemari kunobva mashoko inounganidzwa kubva Twitter (Jordan 2010) . Uchapupu hupi ungada kuona asati kuisa mari yako kuti homwe?

  3. [ nyore ] Kunyange zvazvo vamwe utano vanyatsowana pachena chimvuramabwe e-midzanga kuyamura zvakabatsirawo kusvuta kumbomira, vamwe zvinonyevera nezvengozi dzingangoitika nengozi, akadai mukuru-siyana chikakananda. Fungidzira kuti mutsvakurudzi akasarudza kudzidza nevanhuwo vakatarira e-midzanga nekunhonga e-midzanga-chokuita Twitter mbiru uye kuitisa manzwiro kuongorora.

    1. Ndezvipi zvitatu zvinogoneka vakanga vaine kwavakarerekera kuti uri vakawanda vanonetseka muchidzidzo ichi?
    2. Clark et al. (2016) akamhanya chete kudzidza kwakadaro. Chokutanga, vakaunganidza 850,000 Tweets kuti vakashandisa Keywords e-mudzanga-chokuita kubva January 2012 kuburikidza December 2014. Pamusoro ukanyatsozvitarisa, vakaziva kuti vakawanda Tweets aya zvoga (kureva, kwete kugadzirwa nevanhu) uye vazhinji izvi zvoga Tweets vakanga vanotova zvokushambadza. Ivo yakada Human Detective Algorithm kuti vaparadzane zvoga Tweets kubva makemikari Tweets. Kushandisa Human ichi kuona Algorithm vakawana kuti 80% yevanhu Tweets vakanga zvoga. Ko Kuwana ichi kushandura mhinduro yako chikamu (a)?
    3. Vakati akaenzanisa naA mune makemikari uye zvoga Tweets vakaona kuti zvoga Tweets zvakawanda zvakanaka pane makemikari Tweets (6.17 maringe 5.84). Ko Kuwana ichi kushandura mhinduro yako (b)?

  4. [ nyore ] Muna November 2009, Twitter zvakachinja mubvunzo iri Tweet bhokisi kubva "Muri kuitei?" Kuti "Chii chiri kuitika?" (Https://blog.twitter.com/2009/whats-happening).

    1. Unofunga shanduko runoita kuchachinja ndiani tweet uye / kana zvavanenge tweet?
    2. Doma imwe tsvakurudzo chirongwa icho ungadai usada kuita "Muri kuitei?" Tsanangura nei.
    3. Doma imwe tsvakurudzo chirongwa icho ungadai usada chiriporipocho "Chii chiri kuitika?" Tsanangura nei.

  5. [ nzira ] Kwak et al. (2010) waongorora mamiriyoni 41.7 User Profiles, mabhiriyoni 1.47 magariro ukama, 4262 Trending nyaya, uye mamiriyoni 106 Tweets pakati June 6th uye June yechi31, 2009. Kubva ongororo iyi vakazoona kuti Twitter anoshumira zvakawanda sezvo mutsva svikiro mashoko kugoverana kupfuura social network.

    1. Tichifunga Kwak and said kuti kuwanikwa, chii mhando kutsvakurudza waizoita sei Twitter mashoko? Kworudzii kutsvakurudza haungabvumi kuita Twitter mashoko? Sei?
    2. Muna 2010, Twitter akawedzera Ndiani To Teverai basa kuita yakadekara zviedzwe vanoshandisa. mazano matatu vanoonekwa panguva papeji huru. Zvinotarisirwa vari zvinowanzoratidzwa kubva munhu "shamwari-of-shamwari," uye mose vokukurukura vari rinoratidzirwawo mune kurumbidza kwacho. Users anogona kuzorodza kuona zvitsva vanorumbidza kana kushanyira imwe peji ane refu mazita munzwewo. Munofunga aya matsva aizochinja mhinduro yako chikamu a)? Sei uchidaro kana kusadoro?
    3. Su, Sharma, and Goel (2016) dzinoongororwa migumisiro Who To Teverai basa uye vakawana kuti apo vanoshandisa mhiri kukurumbira chakarondedzera vakabatsirwa kunenge, zvikuru nevanhu vanoshandisa unobatsirwei substantially kupfuura avhareji. Ko Kuwana ichi kushandura mhinduro yako chikamu b)? Sei uchidaro kana kusadoro?

  6. [ nyore ] "Retweet" anowanzoshandiswa kuyera simba uye kupararira simba pamusoro Twitter. Pakutanga, vanozvishandisa vaifanira kutevedzera namatidza ari Tweet chavaida, Tag chepakutanga munyori ake / her mubato, uye manually kutaipa "RT" pamberi Tweet kuratidza kuti ari retweets. Zvadaro, muna 2009 Twitter akawedzera "Retweet" bhatani. Muna June 2016, Twitter akaita kuti vanoshandisa Retweet Tweets yavo (https://twitter.com/twitter/status/742749353689780224). Munofunga nokuchinja uku kunofanira kuchinja sei Unoshandisa "retweet" pakutsvakurudza rako? Sei uchidaro kana kusadoro?

  7. [ nzira , date muunganidzwa , kunoda Coding ] Michel et al. (2011) akavaka corpus zvichiitika kubva Google achiedza digitize mabhuku. Uchishandisa wokutanga shanduro corpus, yakabudiswa muna 2009 uye yaiva pamusoro mabhuku mamiriyoni 5 digitized, hachishandi dzakaongorora shoko kushandiswa kakawanda kuongorora kwomutauro kuchinja uye tsika kuitika. Nokukurumidza Google Books Corpus akava mashoko akakurumbira mabviro vatsvakurudzi, uye 2nd shanduro Database yakabudiswa muna 2012.

    Zvisinei, Pechenick, Danforth, and Dodds (2015) akanyevera kuti vaongorori vanofanira zvakazara munguva zvakabudiswa kwacho ari corpus asati kurishandisa wokucheresa pahupamhi mhedziso. Nyaya huru ndechokuti corpus zviri muraibhurari-akafanana, rine imwe bhuku rimwe nerimwe. Somugumisiro, mumwe, iwane munyori anokwanisa zvinoonekwa pinza mashoko matsva kupinda Google Books duramazwi. Uyezve, Magwaro kwesayenzi vanoumba kuwedzera, zvinhu chikamu corpus panyika 1900s. Uyezve, nokuenzanisa shanduro mbiri Chirungu Ngano datasets, Pechenick bv unto. vakawana uchapupu hunoratidza kuti kushomeka mupise waishandiswa pakugadzira yokutanga shanduro. Vose mashoko zvinodiwa basa inowanika pano: http://storage.googleapis.com/books/ngrams/books/datasetsv2.html

    1. In Michel and said. Chepakutanga mapepa (2011) , vakashandisa 1st shanduro Chirungu mashoko yakatarwa, vakarangana kakawanda yemataurire makore "1880", "1912" uye "1973", uye akagumisa kuti "tiri ndinokanganwa hwedu kare nokukurumidza rega gore "(Fig. 3A, Michel bv unto.). Kutevedzera chete zano uchishandisa 1) 1st shanduro corpus, Chirungu dataset (zvakafanana Fig. 3A, Michel bv unto.)
    2. Zvino kutevedzera chete zano pamwe 1st shanduro, Chirungu yokunyepera dataset.
    3. Zvino kutevedzera chete zano pamwe 2nd shanduro corpus, Chirungu dataset.
    4. Pakupedzisira, kutevedzera chete zano pamwe 2nd shanduro, Chirungu yokunyepera dataset.
    5. Tsanangura kusiyana uye kufanana pakati izvi kwokurangana mana. Unobvumirana Michel bv unto. Chepakutanga kududzirwa kuchengetwa muitiro? (Kuruma nzeve: c) uye d) vanofanira kuva zvakafanana Figure 16 Pechenick bv unto).
    6. Zvino zvamakabvisa replicated uyu Kuwana uchishandisa akasiyana Google Books corpora, sarudza mumwe kuchinja mitauro kana tsika mananara anotaurwa muna Michel bv unto. Chepakutanga mapepa. Unobvumirana dudziro wavo chiedza zvausingakwanisi dzinotaurwa Pechenick bv unto.? Kuti kwenyu nharo hwakasimba, edza kutevedzera chete girafu kushandisa shanduro dzakasiyana-siyana mashoko wakaiswa sezvo kumusoro.

  8. [ zvakaoma , date muunganidzwa , kunoda Coding , wangu kufarira ] Penney (2016) rinoongorora kana kwakapararira kuparidzira pamusoro NSA / PRISM zvokuongorora (kureva Snowden zvakazarurwa) muna June 2013 inobatanidzwa unopinza uye pakarepo vasanyanye motokari kuna Wikipedia nyaya nyaya kuti kusimudza zvakavanzika zvaimunetsa. Kana zvakadaro, kuchinja uku maitiro yaizova kupindirana ane anotyisa chaizvoizvo nemhaka vakawanda nokuongororwa. Kuswedera pedyo Penney (2016) unowanzonzi imwe akambomiswa nguva nhevedzano magadzirirwo uye une chokuita Nzira muchitsauko pamusoro approximating kuedza kubva observational Data (Section 2.4.3).

    Kusarudza nyaya Keywords, Penney akataura ndandanda rakashandiswa US Department of Kumusha Security kuti club uye vachiongorora magariro nevezvenhau. The DHS Pamazita categorizes mamwe mashoko unoongorora zvakawanda nyaya, kureva "Health Concern," "Infrastructure Security," uye "Terrorism." Nokuti boka kudzidzawo Penney akashandisa makumi mana namasere Keywords chokuita "Ugandanga" (ona Table 8 Okuwedzera). Akabva aggregated Wikipedia Nyaya maonero hazvibatsiri iri pamwedzi zvikonzero zvakafanana makumi mana namasere Wikipedia nyaya kwenguva mwedzi matatu nezviviri, kubva pakutanga January 2012 kusvika kumugumo August 2014. Kuti usimbise nharo wake, naiye akasika anoverengeka kuenzanisa mapoka kuburikidza club nyaya maonero nedzimwe.

    Zvino, muri kuenda kutevedzera nokurebesa Penney (2016) . All yakatsvuka mashoko kuti uchada kuti basa iri Inowanika Wikipedia (https://dumps.wikimedia.org/other/pagecounts-raw/). Kana unogona kuwana kubva R pasuru wikipediatrend (Meissner and Team 2016) . Kana ukanyora-up mhinduro dzako, tapota cherechedza izvo date panobva zvawaimboita. (Note: Uyu basa rinowanikawo Chapter 6)

    1. Read Penney (2016) uye kutevedzera Figure 2 izvo zvinoratidza peji maonero kuti "Ugandanga" -related mapeji pamberi uye mushure Snowden kuzarurirwa. Kududzira nevatsvakurudzi.
    2. Zvadaro, kutevedzera Pomuonde 4a, izvo rinoenzanisa boka kudzidzwa ( "Ugandanga" -related nyaya) neboka comparator kushandisa mazwi akadzama muboka pasi "DHS & Other Agencies" kubva DHS nyora (ona Appendikisi Table 10). Kududzira nevatsvakurudzi.
    3. Muchikamu b) iwe akaenzanisa boka kudzidza mumwe boka comparator. Penney rinofananidzwawo mamwe mapoka maviri comparator: "Infrastructure Security" -related nyaya (Okuwedzera Table 11) uye yakakurumbira Wikipedia mapeji (Okuwedzera Table 12). Kwira ane comparator boka dzimwe uye edzai kana zvakawanwa kubva chikamu b) inonyatsoziva mukusarudza boka comparator. Ndoupi kusarudza boka comparator anoita vakawanda musoro? Sei?
    4. Munyori akataura kuti Keywords chokuita "Ugandanga" aishandiswa kusarudza Wikipedia nyaya nokuti hurumende US yataurwa ugandanga sechinhu chinokosha kururamiswa hwayo paIndaneti nokuongororwa miitiro. Sezvo cheki ava 48 "Ugandanga" -related Keywords, Penney (2016) Uyewo akaongorora musi MTurk muchikumbira vakabvunzwa kuti havanyanyi nomumwe Keywords tichitarisa Government Yenhamo, Privacy dzinoshandiswa, uye kudzivirira (Okuwedzera Table 7 uye 8). Kutevedzera mashoko ongororo pamusoro MTurk uye enzanisa migumisiro yenyu.
    5. Pachishandiswa kunoguma chikamu d) uye yenyu kuverenga nyaya, Unobvumirana yemunyori kusarudza nyaya Keywords muboka kudzidza? Sei uchidaro kana kusadoro? Kana kwete, waizoita zvinoratidza pachinzvimbo?

  9. [ nyore ] Efrati (2016) inoti yakavakirwa zvakavanzika mashoko, kuti "mugove zvachose" on Facebook akanga kutsauka munenge 5,5% gore pamusoro gore apo "yepakutanga nhepfenyuro kugovana" raiva pasi 21% gore pamusoro gore. kuderera uku ainyanyoda chazvo pamwe Facebook vanoshandisa pasi makore 30. Mushumo aikonzera kuderera kusvikira zviviri. Imwe kukura kwenhamba "shamwari" vanhu vane on Facebook. Mamwe ndechokuti vamwe kugoverana webasa zvava mameseji uye vakwikwidzi akadai SnapChat. Mushumo akazivisawo mazano akawanda Facebook vakaedza vaonekere kugoverana, kusanganisira News Feed algorithm tweaks kuti yepakutanga namagwatidziro zvikuru, uyewo nguva nenguva zviyeuchidzo yepakutanga namagwatidziro vanozvishandisa "On Day Iri" akawanda apfuura. Ndezvipi zvinoratidzei kana, anoita zvakawanwa izvi nokuda vanotsvakurudza vanoda kushandisa Facebook somunhu date kwaibva?

  10. [ nzira ] Tumasjan et al. (2010) rakashuma kuti kuwanda Tweets kutaura rezvematongerwo enyika vachiwirirana kuwanda mavhoti Bato yakagamuchirwa musarudzo German eparamende 2009 (Figure 2,9). Nemamwe mashoko, akazviratidza kuti unogona kushandisa Twitter kufanotaura sarudzo. Panguva yokudzidza uyu yakabudiswa zvaionekwa chaizvo zvinofadza nokuti zvairatidzika sokuti kushandiswa inokosha nevakawanda tsime mashoko makuru.

    Tichifunga zviri dzakaipa mashoko makuru, zvisinei, unofanira pakarepo kuva chokwadi ichi mugumisiro. VeGermany musi Twitter muna 2009 vakanga chaizvo asiri mumiririri boka, uye vatsigiri rimwe bato kuti tweet pamusoro zvematongerwo enyika kakawanda. Saka, zvinoita sokuti zvinoshamisa kuti zvose zvinobvira vakanga vaine kwavakarerekera kuti unogona kufungidzira zvingadai zvimwe kudzima. Kutaura zvazviri, migumisiro iri Tumasjan et al. (2010) akava kungorotawo kuva zvechokwadi. In mapepa avo, Tumasjan et al. (2010) vaiona nhanhatu mapato: Democrats vechiKristu (CDU), Christian Social Democrats (CSU), SPD, Liberals (FDP), The Left (Die Linke), uye Green Party (green). Zvisinei, chinonyanya kutaurwa German bato musi Twitter panguva iyoyo yaiva Pirate Party (Piraten), imwe pati kuti anorwa hurumende murau Indaneti. Apo Pirate Party waibatanidzwa pachisara Twitter anotaura rava zvinotyisa kuzoita zvabuda musarudzo (Figure 2,9) (Jungherr, Jürgens, and Schoen 2012) .

    Akasabvumirana 2.9: Twitter inotaura sokuti kufanotaura migumisiro 2009 sarudzo German (Tumasjan bv unto 2010.), Asi ichi mugumisiro anozova kuvimba nevamwe kuchengera uye sakodzera sarudzo (Jungherr, Jürgens, uye Schoen 2012).

    Akasabvumirana 2.9: Twitter inotaura sokuti kufanotaura migumisiro 2009 sarudzo German (Tumasjan et al. 2010) , Asi ichi mugumisiro anozova kuvimba nevamwe kuchengera uye sakodzera sarudzo (Jungherr, Jürgens, and Schoen 2012) .

    Zvadaro, dzimwe vatsvakurudzi munyika vakashandisa nzira-zvakadai fancier sezvo kushandisa manzwiro kuongorora kusiyanisa zvakanaka nezvakaipa rinotaura vemapato-kuitira kuvandudza mano Twitter mashoko kufanoziva-siyana emhando dzakasiyana sarudzo (Gayo-Avello 2013; Jungherr 2015, Ch. 7.) . Heano mazano Huberty (2015) akapfupikisa migumisiro kuedza izvi kufanotaura sarudzo:

    "Nzira All kufungidzira anozivikanwa yakavakirwa pasocial nhau vakundikana kana pasi zvinodiwa vechokwadi ramangwana yakanangana nesarudzo kufungidzira. Kukundikana uku kuonekwa kuti zvakafanira kuti anokosha ehupfumi munzanga nhau, pane kuti methodological kana algorithmic matambudziko. Muchidimbu, magariro nhau musaita, uye zvichida haazombobvumiri Mupe kugadzikana, unbiased, mumiririri Mufananidzo electorate; uye kurerukirwa mimwe evanhu nhau kushaya mashoko akakwana kugadzirisa matambudziko aya kuposita hoc. "

    Verenga zvimwe kutsvakurudza atungamirire Huberty (2015) kuti mhedziso, uye kunyora peji imwe dzemari kuti mumiriri enyika achirondedzera kana uye sei Twitter rinofanira kushandiswa ramangwana sarudzo.

  11. [ nzira ] Musiyano pakati nyanzvi uye munyori wenhoroondo? Maererano Goldthorpe (1991) , musiyano mukuru pakati nyanzvi uye munyori wenhoroondo ndiyo kudzora mashoko muunganidzwa. Vezvenhau vanomanikidzwa kushandisa zvisaririra nepo dzezvemagariro evanhu vanogona kuchinja yavo date muunganidzwa kusaruraude zvinangwa. Read Goldthorpe (1991) . Sei musiyano pakati wemagariro evanhu uye nhoroondo chokuita pfungwa Custommades uye Readymades?

  12. [ zvakaoma ] Kuvakira yapfuura mubvunzo, Goldthorpe (1991) akabwembura dzinoverengeka dzinonetsa mhinduro, kusanganisira mumwe achibva Nicky Hart (1994) kuti akapikisa kuzvipira Goldthorpe kuti titaure akaita mashoko. Kujekesa zvinogona kuitwa zvakakukodzera data, Hart akarondedzera vakabudirira Worker Project, hombe kuongorora kuyera ukama pakati munzanga uye kuvhota kuti kwaitungamirirwa Goldthorpe uye nevamwe vari epakati Kuma1960. Somunhu anogona kutarisira kubva nyanzvi vaitsigira yakagadzirwa mashoko pamusoro akawana data, vanhu vakabudirira Worker Project dzakaunganidza mashoko kuti yakanyatsogadzirirwa agadzirise ichangobva aronga Pfungwa nezveramangwana munzanga panguva kuwedzera mararamiro. Asi, Goldthorpe nevaaishanda navo zvimwe "vakakanganwa" kuunganidza mashoko pamusoro kuvhota maitiro wevakadzi. Heano mazano Nicky Hart (1994) nokupeta ndataura yose:

    ". . . izvozvo [ndiko] zvakaoma kudzivisa mhedziso kuti vakadzi vakanga akadarikira nokuti ichi 'musoni akaita' dataset akanga akasungwa ne paradigmatic pfungwa iyo dziripowo hadzi ruzivo. Rinosundwa yokungotaura chiratidzo mukirasi kuziva uye kuita sezvo varume preoccupations. . . , Goldthorpe nevamwe vake kuvakwa kutaridza empirical uchapupu izvo zvokudya nokurerwa pachavo yokungotaura zvekungofungira yavo pane kufumura kuti chikonzero muedzo adequacy. "

    Hart mberi:

    "The empirical Zvakawanikwa ari vakabudirira Worker Project inotiudzei zvakawanda pamusoro masculinist tsika pakati remakore wemagariro evanhu kupfuura ivo udza dzinotsanagura stratification, zvematongerwo enyika uye zvinhu upenyu."

    Ungafunga dzimwe mienzaniso apo zvakakukodzera umbowo muunganidzwa ane vakanga vaine kwavakarerekera humboo nomuteresi yakavakwa mariri? Izvi zvakafanana sei kuti algorithmic akakunda? Ndezvipi zvinoratidzei iyi apo vatsvakurudzi vanofanira kushandisa Readymades uye kana vanofanira kushandisa Custommades?

  13. [ nzira ] Muchitsauko chino, ini Akasiyanisa Data unganidzwa nokuda vaongorori nokuti vanotsvakurudza vane administrative zvinyorwa zvakasikwa makambani nehurumende. Vamwe vanhu vanodana zvinyorwa izvi administrative "vakawana data," izvo ivo Kusiyana "akagadzirirwa mashoko." Ichokwadi kuti administrative zvinyorwa zvinowanikwa nenzira nevatsvakurudzi, asi ivo vari zvikuru yakagadzirwa. Somuenzaniso, mazuva ano Tech makambani anopedza yakawanda nguva nezvinhu kuunganidza uye rakarurama mashoko avo. Saka, zvinyorwa izvi administrative vari vaviri vakawana uye rakagadzirirwa, naro chete kunoenderana maonero ako (Figure 2.10).

    Figure 2.10: The mufananidzo uri zvose dhadha uye netsuro; zvaunoona kunoenderana maonero ako. Hurumende uye mabhizimisi administrative zvinyorwa vari vaviri vakawana uye rakagadzirirwa; zvaunoona kunoenderana maonero ako. Somuenzaniso, kudana Data zvinyorwa unganidzwa nokuda kambani foni zvinowanikwa mashoko nemaonero mutsvakurudzi. Asi, izvi chaiwo chete zvinyorwa yakagadzirirwa date nemaonero munhu kushanda Billing dhipatimendi wekambani foni. Source: Wikimedia Commons

    Figure 2.10: The mufananidzo uri zvose dhadha uye netsuro; zvaunoona kunoenderana maonero ako. Hurumende uye mabhizimisi administrative zvinyorwa vari vaviri vakawana uye rakagadzirirwa; zvaunoona kunoenderana maonero ako. Somuenzaniso, kudana Data zvinyorwa unganidzwa nokuda kambani foni zvinowanikwa mashoko nemaonero mutsvakurudzi. Asi, izvi chaiwo chete zvinyorwa yakagadzirirwa date nemaonero munhu kushanda Billing dhipatimendi wekambani foni. Source: Wikimedia Commons

    Huchapa muenzaniso date panobva apo achizviona zvose zvinowanikwa uye rakagadzirirwa zvinobatsira kana kushandisa kuti mashoko kwaibva pakutsvakurudza.

  14. [ nyore ] Mune aifungisisa rondedzero, Christian Sandvig uye Eszter Hargittai (2015) tsanangura mhando mbiri digitaalinen tsvakurudzo, apo kudigita ndiye "mudziyo" kana "chinhu kudzidza." Muenzaniso kudzidza mutsa yekutanga apo Bengtsson uye nevamwe (2011) rakashandiswa runharembozha Data kuti chamber vaitamira pashure kwenyika kuHaiti muna 2010. muenzaniso wechipiri mhando ndiko Jensen (2007) zvidzidzo sei kutanga dzeserura yose Kerala, India rinoitei kushanda kumusika kuti hove. Ndinowana ichi anobatsira nokuti rinojekesa kuti zvidzidzo vachishandisa digitaalinen mashoko anogona kuva nezvinangwa zvakasiyana kunyange kana vari kushandisa vakadarovo date kunobva. Kuti kujekesa mutsauko ichi, tsanangura zvidzidzo zvina kuti iwe ndamboona: maviri kuti kushandisa kudigita semudziyo uye vaviri kuti kushandisa kudigita sezvo chinhu kudzidza. Unogona kushandisa mienzaniso kubva chitsauko ichi kana uchida.

Powered by Open Review Toolkit