2.3.1 Big

Datasets Large inzira kupera; havasi mugumo pachazvo.

Nyaya inonyanya kukurukurwa pamusoro peshoko guru rekushandisa ndeyokuti iBIG. Mapepa akawanda, semuenzaniso, anotanga nekukurukurirana-uye dzimwe nguva kuzvikudza-pamusoro pehuwandu hwemashoko avakaongorora. Semuenzaniso, pepa rakabudiswa muScience kuverenga mazwi-kushandiswa kwemaitiro muGoogle Books corpus yakaisanganisira zvinotevera (Michel et al. 2011) :

"[Our] corpus ine mashoko anodarika mabhiriyoni mashanu, muchiRungu (361 mabhiriyoni), chiFrench (45 mabhiriyoni), chiGerman (37 mabhiriyoni), chiChina (13 mabhiriyoni), Russian (35 mabhiriyoni), uye rechiHebheru (2 biliyoni). Mabasa ekare akabudiswa muma1500s. Makore emakore ekutanga anomiririrwa nemabhuku mashomanana pagore, ane mazana mazana emakore mazwi. Pakazosvika 1800, corpus inokura kusvika pamamiriyoni 98 mazwi pagore; ne1900, 1.8 mabhiriyoni; uye ne2000, 11 mabhiriyoni. Izvozvo hazvikwanisi kuverengwa nemunhu. Kana ukaedza kuverenga zvinyorwa zveChirungu chete kubva mugore ra2000 roga, panguva yakakodzera yemashoko 200 / min, pasina kukanganisa kwekudya kana kurara, zvinotora makore makumi masere. Kuenzanisa kwemashoko kunopera 1000 zvakapfuvura kupfuura gome revanhu: Kana iwe wakanyora iri mumutsara wakarurama, inosvika kuMwedzi uye kudzoka katatu pamusoro. "

Nhamba ye data iyi hapana mubvunzo inoshamisa, uye isu tose tine nhamo kuti Google Book team yakabudisa idzi idzi kuvanhu (zvirokwazvo, mamwe emabasa ekuguma kwechitsauko chino vanoshandisa deta iyi). Asi, pose paunoona chimwe chinhu chakadai iwe unofanirwa kubvunza: ndechokuti iyo yose data inonyatsoita chero chinhu here? Vangadai vakaita tsvakurudzo imwe chete kana data inogona kusvika kuMwedzi uye inodzoka kamwe chete here? Ko kana iyo data ingangosvika kumusoro kweGomo reEverest kana pamusoro peEiffel Tower?

Muchiitiko ichi, kutsvakurudza kwavo, ikoizvo, kune zvimwe zvitsva zvinoda kubatana kukuru kwemashoko kwenguva refu. Somuenzaniso, chimwe chinhu chavanotsvaga ndechokushanduka kwegirama, kunyanya kuchinja mumutengo wezvisinganzwisisiki. Sezvo mamwe mazita asinganzwisisiki asingawanzoviki, ruzhinji rwemashoko rinodikanwa kuona kuchinja kwemazuva. Kazhinji, zvisinei, vanotsvakurudza vanoita sevanoshandisa hukuru hwemashoko makuru ekupedzisira sekumagumo- "tarisa kuti inguva yakawanda sei ini ndinogona kuisa" -kusanganisa nzira yeimwe chinangwa chinokosha chezesayenzi.

Muhupenyu hwangu, kudzidza kwezviitiko zvisingawanzo ndeimwe yezvinhu zvitatu zvakanyatsozivikanwa nesayenzi iyo huru dzemasitadhi dzinowanzoita kuti dzigone. Chechipiri ndechidzidzo chehutano hwakanaka, sezvinogona kufananidzwa nekudzidza naRaj Chetty uye vashandi (2014) pamusoro pekugara zvakanaka kwevanhu muUnited States. Munguva yakapfuura, vatsvakurudzi vakawanda vakaongorora kusununguka kwevanhu nekuenzanisa zvibereko zveupenyu zvevabereki nevana. Kuwana kusingagumi kubva kune mabhuku aya ndeyekuti vabereki vakabudirira vanofarira kuva nevana vakanaka, asi simba rehukama hunosiyana nekufamba kwenguva uye nenyika dzose (Hout and DiPrete 2006) . Munguva pfupi yapfuura, zvakadaro, Chetty nevamwe vashandi vakakwanisa kushandisa zvinyorwa zvemitero kubva kuvanhu vane mamiriyoni makumi mana kuti vaongorore hutano hwakanaka mumasangano evanhu vanoenda kune dzimwe nzvimbo muUnited States (chirevo 2.1). Vakatora, somuenzaniso, kuti mukana wekuti mwana anosvika nepamusoro ye quintile yekugoverwa kwemari yenyika kubva kumhuri iri pasi ye quintile inenge 13% muSan Jose, California, asi inenge 4% chete muCharlotte, North Carolina. Kana iwe ukatarisa chimiro 2.1 kwekanguva, unogona kutanga kushamisika kuti nei kuparadzaniswa kwevanhu vari pakati kune dzimwe nzvimbo kupfuura vamwe. Chetty uye vashandi vakanga vane mubvunzo wakafanana, uye vakawana kuti iyo nzvimbo dzakakwirira-dzika dzine uwandu hushoma hwekugara, kusagadzikana kwepasuo, zvidzidzo zvidzidzo zvepamusoro, hukuru hwemagariro evanhu, uye hukama hwakasimba hwemhuri. Ichokwadi kuti kuwirirana uku chete hakuratidzi kuti izvi zvinokonzera kukwirira kwepamusoro, asi zvinoratidza nzira inogona kuwanikwa inogona kuongororwa mune mamwe mabasa, izvo chaizvoizvo Chetty nevamwe vashandi vakaita mubasa rinotevera. Cherechedza kuti hukuru hwe data hwakakosha sei mujekiti iyi. Kana Chetty uye vashandi pamwe chete vakashandisa zvinyorwa zvemitero zvevanhu zviuru makumi mana pane mamiriyoni makumi mana, vangadai vasina kukwanisa kutarisa hutano hwenyika uye havazombokwanisi kuita tsvakurudzo inotevera kuti vaedze kuona nzira dzinoita kuti zvivepo.

Mufananidzo 2.1: Inofanirwa nemikana yevana yekusvika 20% yepamusoro yekugovera mari yakapiwa vabereki muzasi 20% (Chetty et al. 2014) . Nharaunda dzenharaunda-iyo, iyo inoratidza huterogeneity, zvinotungamirira kune mibvunzo inofadza uye inokosha isingasviki kubva kune imwe yenyika-yakatarwa. Izvi zvikwereti zvepamatunhu zvakagadziridzwa muchidimbu nokuti vatsvakurudzi vakanga vari kushandisa guru guru rekushandisa dhiyabhorosi: marekodha emitero emamiriyoni makumi mana evanhu. Yakagadzirwa kubva kune data iripo pa http://www.equality-of-opportunity.org/.

Pakupedzisira, kunze kwekudzidza zviitiko zvisingawanzoitiki nekudzidza heterogeneity, huru dzemasitetiti dzinobatsirawo vatsvakurudzi kuti vaone zviduku zvakasiyana. Zvechokwadi, zvizhinji zvekutarisa kune dheinhau guru mune zvemabhizimisi ndezvekusiyana kweizvi zvishoma: kutendeseka nekuona mutsauko pakati pe1% ne1 1.1% kudarika-kuburikidza nemitengo pamusangano unogona kushandura mumamiriyoni emadhora mune mamwe mari. Mune mamwe masayendisiti akaiswa, zvakadaro, kusiyana kwakadaro kuduku hakugone kuve kwakakosha, kunyange kana vari nhamba inokosha (Prentice and Miller 1992) . Asi, mune dzimwe zvirongwa zvepurogiramu, dzinogona kuva dzinokosha kana dzichionekwa muzvikamu. Semuenzaniso, kana kune huviri hwehutano hwehutano uye humwe humwe hunobudirira kupfuura humwe, ipapo kuunganidza kupindira kwakanyatsokwanisa kunogona kupedzisira kuponesa zviuru zvehupenyu humwe.

Kunyange zvazvo hukuru huri humbo hwakanaka kana huchishandiswa zvakanaka, ndakacherechedza kuti dzimwe nguva inogona kutungamirira kune pfungwa yekukanganisa. Nokuda kwechimwe chikonzero, hukuru hunofanirwa kutungamirira vanotsvakurudza kusava nehanya kuti data yavo yakagadzirwa sei. Kunyange hukuru huri kuderedza kudiwa kwekushushikana nezvisina kukanganisa, zvinowedzera kukosha kwekushushikana pamusoro pezvakanganisa zvikanganiso, mhando dzekanganiso dzandicharondedzera pasi apa dzinomuka kubva mukusavhiringidza kwokuti data yakasikwa sei. Somuenzaniso, mune chirongwa ndicharondedzera gare gare muchitsauko ichi, vatsvakurudzi vakashandisa mharidzo yakagadzirwa musi waSeptember 11, 2001 kuti ibudise nguva yakagadzirisa mafungiro emagariro ezvakaitika kune hugandanga (Back, Küfner, and Egloff 2010) . Nemhaka yekuti vatsvakurudzi vaiva nehuwandu hwemashoko, ivo vaisafanira kunyanya kunetseka pamusoro pekuti mararamiro avakaona-kuwedzera kutsamwira pamusoro penzira yezuva-kunogona kutsanangurwa nekusiyana-siyana. Kwakanga kune deta yakawanda uye iyo muenzaniso yakanga yakanyatsojeka zvokuti dzose nhamba dzematanho emitambo dzakaratidza kuti iyi yaiva yechokwadi chaiyo. Asi, izvi zvinongororwa zvidzidzo zvakanga zvisingazivi kuti data yakasikwa sei. Muzvokwadi, zvakazoitika kuti dzakawanda zvepurogiramu dzaive dzakabatanidzwa kune rimwe bhodhi rakagadzira mharidzo yakawanda uye isina chinangwa zuva rese. Kubvisa iyi bot zvakaparadzwa zvachose zvimwe zvitsva zviri mupepa (Pury 2011; Back, Küfner, and Egloff 2011) . Zviri nyore, vatsvakurudzi vasingafungi nezvekugadzirisa kukanganisa vanotarisana nedambudziko rekushandisa dhemasti makuru avo kuti vawane kuongororwa kwakajeka kwehuwandu husingakoshi, hwakadai sehuwandu hwemagariro emashoko asingabatsiri anogadzirwa ne-automated bot.

Mukupedzisa, mabhizimusi makuru haasi magumo mavari, asi anokwanisa kuita mamwe marudzi ekutsvakurudza kusanganisira kuongorora zviitiko zvisingashamisi, kuongororwa kwehutano hwakanaka, uye kuonekwa kwezvishoma zvisiyana. Dhiyabhorosi makuru anotaridzikawo kutungamirira vamwe vatsvakurudzi kuti varege kunyatsoziva kuti deta yavo yakasikwa sei, inogona kuvatungamirira kuwana kuongororwa kwakajeka kwehuwandu husingaverengeki.