A Hadoop based Web log Analysis System

$+DGRRSEDVHG:HEORJ$QDO\VLV6\VWHP

&KHQ+DX:DQJ

&KLQJ7VRUQJ7VDL

&KLD&KHQ)DQ

6K\DQ0LQJ<XDQ

'HSDUWPHQWRI&RPSXWHU

6FLHQFH



'HSDUWPHQWRI&RPSXWHU

6FLHQFH

'HSDUWPHQWRI&RPSXWHU

6FLHQFH

1DWLRQDO&KLDR7XQJ

8QLYHUVLW\

7XQJKDLXQLYHUVLW\

1DWLRQDO&KLDR7XQJ

8QLYHUVLW\

1DWLRQDO&KLDR7XQJ

8QLYHUVLW\

+VLQFKX7DLZDQ

7DLFKXQJ7DLZDQ

+VLQFKX7DLZDQ

+VLQFKX7DLZDQ

FKZDQJ#JPDLOFRP

FWWVDL#WKXHGXWZ

ZDQG\#\DKRRFRPWZ

VP\XDQ#FVQFWXHGXWZ



LPSRUWDQW LVVXH LQ WKH ILHOG RI UHVHDUFK &ORXG FRPSXWLQJ HPSOR\V GLVWULEXWHG VWRUDJH DQG GLVWULEXWHG FRPSXWLQJ WHFKQRORJ\ WR DFKLHYH D ODUJHQXPEHU RI VWRUHG GDWD DVZHOO DV IDVW GDWD DQDO\VLV DQG SURFHVVLQJ $V WKH UDSLG GHYHORSPHQW RI ,QWHUQHW WHFKQRORJ\ GLJLWDO GDWD VKRZLQJ H[SORVLYH JURZWK WKH IDFHRIPDVVLYHGDWDSURFHVVLQJWKHWUDGLWLRQDOWH[WVRIWZDUHDQG UHODWLRQDO GDWDEDVH WHFKQRORJ\ KDV EHHQ IDFLQJ D ERWWOHQHFN SUHVHQWHGWKHUHVXOWVDUHQRWYHU\VDWLVIDFWRU\)RUWKLVSUREOHP WKHFRQFHSWRIFORXGFRPSXWLQJLVDPRUHDSSURSULDWHFKRLFH,Q WKLV SDSHU EDVHG RQ WKH DUFKLWHFWXUH RI +DGRRS ZLWK +')6 +DGRRS 'LVWULEXWHG )LOH 6\VWHP DQG +DGRRS 0DS5HGXFH VRIWZDUH IUDPHZRUN DQG 3LJ /DWLQ ODQJXDJH ZH GHVLJQ DQG LPSOHPHQW DQ HQWHUSULVH :HEORJ DQDO\VLV V\VWHP ([SHULPHQWDO UHVXOWV E\ DQDO\]LQJ GDLO\ :HEORJ UHFRUGV ZH JHW $SSOLFDWLRQ 6HUYHUWUDIILFWUHQGVSHUIRUPDQFHRISURJUDPVWDWLVWLFDOUHSRUWV DQG SHUIRUPDQFH UHSRUWV RI GLIIHUHQW LQWHUYDOV DQG GLIIHUHQW DFWLRQV RI SURJUDP E\ XVHU UHTXHVW 7KH PDLQ SXUSRVH RI WKLV V\VWHPLVWRDVVLVWV\VWHPDGPLQLVWUDWRUVWRTXLFNO\FDSWXUHDQG DQDO\]H GDWD KLGGHQ LQ WKH PDVVLYH SRWHQWLDO YDOXH WKXV SURYLGLQJDQLPSRUWDQWEDVLVIRUEXVLQHVVGHFLVLRQV

Keywords—Cloud Computing, Distributed File System, Pig programming language

, ,1752'8&7,21

,Q UHFHQW \HDUV LQWHUQHW DQG PRELOH KDYH EHFRPH YHU\ SXEOLF7KHVHGHYLFHVJHQHUDWHGDORWRIGDWD&ORXG&RPSXWLQJ LVDKRWWRSLFWRGHDOZLWKWKLVLQIRUPDWLRQ&ORXGFRPSXWLQJ WHFKQRORJ\XWLOL]HVWKHLQWHJUDWLRQRIUHVRXUFHVWRUHGXFHFRVWV DQGVLPSOLI\FRPSXWLQJSODWIRUP,WDIIHFWVRXUOLIHDQGZRUN *RRJOH KDV GHYHORSHG D QXPEHU RI FORXG FRPSXWLQJ WHFKQRORJ\DQGDUFKLWHFWXUHVXFKDV0DS5HGXFH*RRJOH)LOH 6\VWHPHWF7KHVHWHFKQRORJLHVDQGDUFKLWHFWXUHKDVDIHDWXUH WKDWGHYHORSHUVGRQ¶WNQRZKRZWRSODFHWKHGDWDDQGFXWWKH FRPSXWLQJRQGLVWULEXWHGV\VWHPV'HYHORSHUVIRFXVKLVPLQG RQVHUYLFHGHYHORSPHQWDQGXQGHUO\LQJDUFKLWHFWXUHKDQGOHWKH GDWD DQG GLVSHUVLQJ DQG FXWWLQJ RI FRPSXWLQJ 7KHVH WHFKQRORJLHV DQG DUFKLWHFWXUHV FDQ LQFUHDVH WKH VSHHG RI

WR JHW EHWWHU DW ZRUNLQJ XS HIILFLHQF\ 5HODWLRQDO 'DWDEDVH 7HFKQRORJ\$UFKLWHFWXUHRIFORXGDUFKLWHFWXUHLVYHU\GLIIHUHQW IURP WUDGLWLRQDO DUFKLWHFWXUH ,W LV GLIILFXOW LQ WKH WHFKQLFDOO\ ZKHQODUJHHQWHUSULVHVPLJUDWHWUDGLWLRQDOGDWDEDVHV\VWHPVWR FORXGEDVHG RU FUHDWLQJ D FORXG FRPSXWLQJ GDWDEDVH :H FDQ RYHUDOO GHVLJQHG DQG FRQVLGHU WKH DGYDQWDJHV RI FORXG FRPSXWLQJWHFKQRORJ\WRLQWHJUDWLRQRIWKHLQWHUQDOUHVRXUFHVLI ZH GHYHORSPHQW D QHZ V\VWHPV 7KH V\VWHP XVHV WKH PRVW HIILFLHQWDQGWKHHDVLHVWPDQDJHWRSURYLGHDWDEOHDQGUHOLDEOH FRPSXWLQJDQGVWRULQJHQHUJ\VHUYLFHV ,, 5(/$7('5(6($5&+$1'%$&.*5281' A. Hadoop

+DGRRS >@ LV WKH RSHQVRXUFH SURJUDP XQGHU $SDFKH 6RIWZDUH )RXQGDWLRQ ,W FRPHV IURP D SDUW RI LQWHUQHW VHDUFK HQJLQHLQ$SDFKH1XWFK>@+DGRRSZDVGHYHORSHGIURPMDYD ,WLVDGLVWULEXWHGFRPSXWLQJVRIWZDUHSODWIRUPWKDWXVHUHDVLHU ZULWLQJDQGKDQGOLQJDORWRIGDWD +DGRRSVRIWZDUHDUFKLWHFWXUHIHDWXUHV ,WFDQKDQGOHDQGVWRUDJHDPRXQWRIGDWD 8VLQJGLVWULEXWHG)LOH6\VWHPREWDLQTXLFNUHVSRQVH  $XWRPDWLFDOO\ REWDLQHG EDFNXS GDWD DQG GHSOR\PHQW RI FRPSXWLQJUHVRXUFHVZKHQWKHHUURURFFXUUHGLQDQRGH

7KH +DGRRS DUFKLWHFWXUH RI FRUH LV +DGRRS 'LVWULEXWLRQ )LOH V\VWHP +')6 >@ DQG *RRJOH 0DS5HGXFH RI RSHQ 6RXUFH>@>@*RRJOHSURSRVHVWKH&ORXGNH\WHFKQRORJ\WR LPSOHPHQW +DGRRS *RRJOH FRPSDUHG ZLWK +DGRRS LV LQ WKH 7$9/( , +')6 FDQ DXWRPDWLF DFTXLVLWLRQ WKH EDFNXS GDWD DQGGHSOR\PHQWRIFRPSXWLQJUHVRXUFHVZKHQWKHQRGHHUURU +DGRRS FDQ GHSOR\ LQ FKHDS KDUGZDUH WR IRUP D GLVWULEXWHG V\VWHP 0DS5HGXFH SURJUDPPLQJ PRGHO DOORZV XVHUV WR DSSOLFDWLRQGHYHORSPHQWZKHQGRQ¶WXQGHUVWDQGWKHXQGHUO\LQJ GHWDLOVRIWKHGLVWULEXWHGV\VWHP

7$%/(, *22*/(&203$5(':,7++$'223

B. HDFS

+')6LVGLVWULEXWHG)LOH6\VWHPLQ+DGRRS,WLVHDVLHUWR H[SDQG 7KLV SXUSRVH LV DQDO\VLV D ORW RI GDWD RSHUDWHV RQ LQH[SHQVLYH KDUGZDUH DQG SURYLGH IDXOW WROHUDQFH +')6 LV PDVWHUVODYH PRGHO DUFKLWHFWXUH 0DVWHU 6ODYH $ VHW RI +DGRRS FOXVWHU LV FRPSRVHG RI D 1DPHQRGH DQG PDQ\ 'DWDQRGH 1DPHQRGH LV PDQDJHPHQW RI WKH ILOH V\VWHP QDPHVSDFH WUHH VWUXFWXUH RI PDLQWHQDQFH WKH ILOH V\VWHP DQG DOOFRQWHQWVLQIRUPDWLRQ)LOHV\VWHPLVVWRUHGLQWKH'DWDQRGH 'DWDQRGH XVXDOO\ VHUYHG E\ D QXPEHU RI GLIIHUHQW QRGHV DQG UHWXUQ WR 1DPHQRGH FXUUHQWO\ VWRUDJH EORFN OLVW LQIRUPDWLRQ +')6LQRUGHUWRHQVXUHGDWDFRQVLVWHQF\VRSURYLGHDVLQJOH GLUHFWRU\V\VWHPDQGILOHQDPHVSDFH7KHDFFHVVPRGHLVRQFH ZULWLQJDQGPDQ\WLPHVDFFHVV$GDWDILOHEHFXWLQWRVHYHUDO VPDOOHUEORFNV>@DQGVWRUHGLQDGLIIHUHQW'DWDQRGHZKHQWKH XVHU ZULWLQJ WKH +')6 UHTXLUHPHQW (DFK EORFN KDV VHYHUDO UHSOLFDWLRQV VWRULQJ LQ GLIIHUHQW QRGHV WR DFKLHYH KLJK IDXOW WROHUDQFHIHDWXUHV7KH+')6DUFKLWHFWXUHLVLQWKH)LJ  )LJ +')6DUFKLWHFWXUH C. MapReduce *RRJOHSURSRVHGDQLPSRUWDQWWHFKQLTXHWRFRPSXWHDORW RI GDWD LV 0DS5HGXFH 7KLV WHFKQLTXH XVHV GLVWULEXWHG FRPSXWLQJ7KHZRUNUHTXLUHVDORWRIFRPSXWLQJUHVRXUFHVFXW LQWRPDQ\VXEZRUNDFURVVPXOWLSOHFRPSXWLQJXQLWDQGILQDOO\ PHUJHWKHUHVXOWVRIPXOWLSOHFRPSXWLQJXQLWIRUWKHILQDOUHVXOW 7KH0DS5HGXFHGDWDIORZDQGFRQWUROIORZLVLQWKH)LJ  )LJ 0DS5HGXFHGDWDIORZDQGFRQWUROIORZ D. Pig

3LJ >@ >@LV 64/OLNH ODQJXDJH ,W LV D KLJKOHYHO GDWD SURFHVVLQJODQJXDJHDQGVXLWDEOHIRUSURFHVVLQJODUJHGDWDVHWV 3LJODQJXDJHLVPRUHIRFXVHGRQGDWDTXHU\DQGDQDO\VLVUDWKHU WKDQWKHGDWDWRPRGLI\DQGGHOHWHRSHUDWLRQV

3LJLVPDLQO\SHUIRUPHGLQDGLVWULEXWHGFORXGSODWIRUP3LJ KDV WKH DGYDQWDJH RI YHU\ KLJK VSHHG FDQ KDQGOH D ORW RI LQIRUPDWLRQLQDVKRUWWLPH6XFKDVV\VWHPORJVSHFLILFZHE GDWDDODUJHGDWDEDVHILOHVHWF

,,, 0(7+2'2/2*<$1',03/(0(17$7,21

A. Weblog System architecture format

7KLVSDSHUXVHGWKHILQDQFHWKHFRPSDQ\VFRUHLQGRPHVWLF 7KH FRUH V\VWHP DUFKLWHFWXUH LV LQ WKH )LJ  7KH EXVLQHVV PRGHOVRIWKHILQDQFLDOVHFWRULVQRWRQO\ZRUNLQJLQWKHRIILFH EXW DOVR YLVLWLQJ RXWVLGH FXVWRPHUV DQG VXSSO\ LPPHGLDWH VHUYLFHV %RWK UHTXLUHPHQWV RI VWDII DQG KDQGOLQJ FXVWRPHU EXVLQHVVDUHXVLQJWKLVFRUHV\VWHP$SSOLFDWLRQVHUYHUFRQWDLQV WZR VHFWLRQV WKDW LQWHUQDO QHWZRUN DQG H[WHUQDO QHWZRUN $Q LQWUDQHW QHWZRUN FRQVLVWV RI WKUHH $SSOLFDWLRQ 6HUYHUV RI FOXVWHUDUFKLWHFWXUHLQ)LJ7KHWKUHH$SSOLFDWLRQ6HUYHUVDUH $3$3DQG$3UHVSHFWLYHO\7KH\KDYHDORDGEDODQFLQJ PHFKDQLVP ([WHUQDO QHWZRUN FRQVLVWV RI WZR $SSOLFDWLRQ 6HUYHUVRIFOXVWHUDUFKLWHFWXUHLQ)LJ7KHWZR$SSOLFDWLRQ 6HUYHUV DUH $3 DQG $3 7KH\ KDYH D ORDGEDODQFLQJ PHFKDQLVP WRR $OO $SSOLFDWLRQ 6HUYHUV DFFHVV WKH VDPH RQH '% 6HUYHU &RQVLGHULQJ LQIRUPDWLRQ VHFXULW\ WKH\ DUHQ¶W GLUHFWO\FRQQHFWHGEHWZHHQ6HUYHUDQG6HUYHU7KH\DUHQHHGHG FRPPXQLFDWLRQWKURXJKFRQWUROOLQJDILUHZDOO



)LJ ([LVWLQJV\VWHPDUFKLWHFWXUH

)LYH $SSOLFDWLRQ 6HUYHUV DUH GRFXPHQWHG DOO SURJUDP H[HFXWLRQ WUDFH LQ WKH )LJ  7KHVH GRFXPHQWV DUH FDOOHG D :HEORJ$:HEORJLVWH[WIRUPDWVDYLQJLQRQHDFKPDFKLQH IRU FRQYHQLHQW DQDO\VLV 7KH EXVLQHVV YROXPH LQFUHDVH VR ZHEORJ YHU\ ODUJH GDLO\ ,W LV YHU\ WLPH FRQVXPLQJ DQG LPSUDFWLFDOWRDQDO\VLVWKHGDWD7KHUHIRUHLWLVYHU\LPSRUWDQW WKDWGHVLJQDV\VWHPWRKDQGOHDQGDQDO\]HGWKHKXJH:HEORJ E\ WKH FKDUDFWHULVWLFV RI FORXG FRPSXWLQJ WHFKQRORJ\ 7KH V\VWHP LV LPSRUWDQW ZKHQ SHUIRUPDQFH WXQLQJ V\VWHP H[SDQVLRQHTXLSPHQWXSJUDGHVRUIXWXUHSURFXUHPHQWSODQQLQJ

B. System environment and parameter settings

7KLV SDSHU XVHV +DGRRS EXLOGLQJ D VHW RI DSSOLFDWLRQ V\VWHP WR VHHNLQJ WKH VROXWLRQ RI ZHEORJ GDWD DQDO\VLV 7KH V\VWHPFRPSLOHVLQWHUQDOQRUPVDQGGRHVQ¶WDIIHFWFRPSDQ\¶V RSHUDWLRQV,WKRSHVWKDWKDYHUHIHUHQFHYDOXHLQHQWHUSULVH,7 DUFKLWHFWXUH 7KH H[SHULPHQWDO DUFKLWHFWXUH VLPXODWHV +DGRRS LQ WKH 90ZDUH HQYLURQPHQWV )LUVW WKH PDLQ LV IXQFWLRQ WHVW DQG SURJUDP IORZ 6HFRQG WKH LQIRUPDWLRQ PRYHV WR UHDO FOXVWHUDUFKLWHFWXUHDIWHUFRQILUPLQJWKHGDWD%HFDXVH+DGRRS LVDQHFRV\VWHPDQGLWFRQWDLQVPDQ\FROODERUDWLYHSURMHFWV,W KDVFRPSDWLELOLW\LVVXHVEHWZHHQGLIIHUHQWYHUVLRQVRIGLIIHUHQW SURMHFW 7KHUHIRUH H[SHULPHQWDO HQYLURQPHQW XVHG LQ VWDEOH YHUVLRQ ([SHULPHQWDO HQYLURQPHQW KDUGZDUH DQG VRIWZDUH VSHFLILFDWLRQVDUHDVIROORZV 90ZDUHYHUVLRQ90ZDUHVHUYHU 1DPHQRGH DQG 'DWDQRGH *% RI 5$0 V\VWHP 8EXQWX DQGNHUQHOYHUVLRQ +DGRRS SODWIRUP XVHG KDGRRS DQG SLJ UHVSHFWLYHO\+DGRRSSDFNDJHSURYLGHVDSURILOHDQGGHVFULEHG LQWKHIROORZLQJ7$%/(,, 7$%/(,, +$'223&21),*85$7,21),/(62)(19,5210(17$/ 3$5$0(7(56 )LOHV ([SODQDWLRQ KDGRRSHQYVK :KHQWKHRSHUDWLRQRI+DGRRSHQYLURQPHQW FRUHVLWH[PO +DGRRSFRUHFRQILJXUDWLRQVXFKDV,2VHWWLQJVRI+')6DQG0DS5HGXFH,3ORFDWLRQRIPDVWHU KGIVVLWH[PO %DFNJURXQGVHUYLFHVHWWLQJVRI+')6WKHQXPEHURI_{QDPHQRGHGDWDQRGHUHSOLFDWLRQ} PDSUGVLWH[PO %DFNJURXQGVHUYLFHVHWWLQJVRI0DS5HGXFH_{MREWUDFNHUVDQGWDVNWUDFNHUV}

PDVWHU ,PSOHPHQWVHFRQGDU\QDPHQRGHPDFKLQHOLVW VODYHV ,PSOHPHQWGDWDQRGHDQGWDVNWUDFNHUPDFKLQHOLVW



+DGRRS FDQ EH VWDUWHG DIWHU FRPSOHWLRQ RI WKH UHOHYDQW VHWWLQJVDQGREVHUYHIXQFWLRQLQJSURSHUO\

,W VWDUWV +DGRRS DQG H[HFXWLRQ RI ELQVWDUWDOOVK LQ WKH LQVWDOODWLRQ GLUHFWRULHV ,W ZLOO RSHQ 1DPHQRGH 'DWDQRGH -RE7UDFNHU 7DVN7UDFNHU UHODWHG LQIRUPDWLRQ LQ WKH SURFHVV :H XVH MSV LQVWUXFWLRQ LQTXLU\ LQWR SURFHVV LG RI HDFK FRPSRQHQW7KH+DGRRSVWDUWVDPSOHLVLQWKHIROORZLQJ)LJ



)LJ +DGRRSVWDUWVDPSOH

7KH LQVWDOODWLRQ RI WKH 3LJ LV YHU\ HDV\ 7KH SUHSDUH -DYD HQYLURQPHQWWKHQWRXQ]LSWKH3LJLQWRSODQQHGSDWKDQGWRVHW UHODWHG 3$7+ LQ HQYLURQPHQW YDULDEOH 7KH 3LJ HQYLURQPHQW YDULDEOH VHWWLQJ LV LQ WKH IROORZLQJ )LJ  3LJ FDQ GLUHFW H[HFXWLRQZKHQWKHEDWFKVFKHGXOLQJRULPPHGLDWHQHHG



)LJ 3LJHQYLURQPHQWYDULDEOHVHWWLQJ

C. System architecture and processes

,QV\VWHPLPSOHPHQWDWLRQDOORIWKHSURIHVVLRQVUHFRUGVDUH GLVSHUVHGLQWKHGLIIHUHQW$3VHUYHU)LUVW$3VHUYHU¶V:HEORJ VSUHDG /LQX[ WKURXJK WKH )73 SURWRFRO GRLQJ SUHOLPLQDU\ FROODWLRQ,QWKHVDPHSURIHVVLRQXQLWVLQWHJUDWHGLQWRDORJILOH VXFK DV )LJ  VWHS D $IWHU WKH ILOHV KDYH ILQLVKHG FXWWLQJ GHVLJQDWLRQEORFNVL]HXSORDGLQWR+')6VSHFLILHGGLUHFWRULHV WKURXJKFRPPDQGVXFKDVIROORZLQJ)LJVWHSE,WGRHVQ¶W FKDQJH WKH GDWD DIWHU :HEORJ ILOH VWRUHV LQ +')6 6\VWHP DGPLQLVWUDWRU GHYHORSV 3LJSURJUDP WR ORDG WKH ORJ DQG FRS\ DQDO\VLVUHVXOWWRUHVWRUHIRU/LQX[GLUHFWRULHVVXFKDVIROORZLQJ )LJ  VWHS F )LQDOO\ 6\VWHP DGPLQLVWUDWRU FUHDWH GLIIHUHQW GLPHQVLRQVDQDO\VLVUHSRUWVDFFRUGLQJUHVXOWVVXFKDVVDOHVVLGH $SSOLFDWLRQ6HUYHUIORZDQGGDWDHWFVXFKDVIROORZLQJ)LJ VWHSG  )LJ 6\VWHPIORZFKDUW ,9 6<67(0,03/(0(17$7,21 A. Program Development 7KHV\VWHPIXQFWLRQVDUHGLYLGHGLQWRWZRFDWHJRULHV,WLV EDWFK DQDO\VLV DQG LQWHUDFWLYH LQSXW FRQGLWLRQV 7KH PDLQ SXUSRVH RI EDWFK DQDO\VLV LV WUDIILF VWDWLVWLFV DQG FRQGXFW PXOWLGLPHQVLRQDO DQDO\VLV ,QWHUDFWLYHIXQFWLRQ LV DQDO\VLV VSHFLILF EXVLQHVV DQG VSHFLILF UDQJH 7KH EDVLF SURJUDP

VWUXFWXUH LV D VKHOO VFULSW IUDPH WR UHFHLYH GLIIHUHQW QHHGV SDUDPHWHUVDQGWRFDOOWKH3LJVFULSW7KHSURJUDPIORZLVLQWKH IROORZLQJ)LJ



)LJ 3URJUDPIORZ

B. System traffic trends Batch Analysis

7KHH[DPSOHLVD:HEORJILOHDW0DUFKWR0DUFK ,WLVVHOHFWHGLPSRUWDQWEXVLQHVVIRUDQDO\VLVREMHFWV7KHUHVXOW LVLQWKHIROORZLQJ

7KH PDLQ SXUSRVH RI GDLO\ V\VWHP WUDIILF VWDWLVWLFV FDQ REVHUYH WKH JURZWK WUHQG RI 6HUYHU ORDGLQJ 7KH GDWD LV WKH LPSRUWDQW LQIRUPDWLRQ RI V\VWHP XSJUDGH RU H[SDQG 7KH IROORZLQJ )LJ  LV WKH DQDO\VLV RI HDUO\ 0DUFK )RU WKLV DQDO\VLV UHVXOW ZH IRXQG D SRWHQWLDO SUREOHP IRU V\VWHP DGPLQLVWUDWLRQ WR UHIHUHQFH RU UHYLHZ WKDW VWDELOLW\ RI ORDG EDODQFLQJPHFKDQLVP



)LJ 'DLO\V\VWHPIORZFKDU

$QRWKHU DQDO\VLV LV REVHUYHG GDLO\ WUDIILF WUHQGV EDVLV RQ HDFK EXVLQHVV VXFK DV IROORZLQJ )LJ %HFDXVH RI WKH FKDUDFWHULVWLFV RI ILQDQFLDO RSHUDWLRQV DQG LQWHUQDO SROLF\ IDFWRUVFDQEHVXPPHGXSVRPHLPSRUWDQWGDWHV6XFKDVWKH GDWD DUH SD\ GHDGOLQH SHUIRUPDQFH EDODQFH VKHHW GDWH DQG SURGXFW SURPRWLRQ 7KH V\VWHP ORDG ZLOO EH VLJQLILFDQWO\ GLIIHUHQWZKHQLPSRUWDQWGDWHV7KHSXUSRVHRI)LJSURYLGHV FRQVLGHUDWLRQV IRU V\VWHPV PDQDJHPHQW RU IXWXUH V\VWHPV DVVHVVPHQWSODQQLQJEHIRUHWKHVSHFLDOGDWD



)LJ 7KHEXVLQHVVIORZFKDUWV

C. System flow specified interval analysis

,QDGGLWLRQWRWKHVWDWLVWLFVRIGDLO\EDWFKJHQHUDWLQJWKHUH DUHWHPSRUDU\QHHGHGIRUTXHU\VSHFLILFLQWHUYDORIV\VWHPIORZ 7KHUHIRUHWKHSDSHUGHYHORSHGDQLQWHUDFWLYHIXQFWLRQRILQSXW FRQGLWLRQV 7KLV IXQFWLRQ LV PDLQO\ SURYLGHG IRU V\VWHP PDQDJHPHQWVXFKDVIROORZLQJ)LJ



)LJ7KHLQSXWVFUHHQRIV\VWHPIORZFRQGLWLRQVVSHFLILHG

D. Program performance statistics

,Q DGGLWLRQ WR IRFXV RQ V\VWHPIORZ GDWD WKH SDSHU LV DQDO\]HGWKHSURJUDPIHDWXUHVVLGH7KHGDLO\EDWFKVWDWLVWLFDO UHSRUWV VKRZ WKH GDLO\ WRWDO QXPEHU RI H[HFXWLRQV IRU HDFK EXVLQHVV SURJUDP VXFK DV WKH 76%/( ,,, DQG WKH DYHUDJH H[HFXWLRQWLPHRIVDOHVSURJUDPVXFKDVWKH7$%/(,9

7$%/(,,, 7+((;(&87,217,0($1'67$7,67,&62)6,1*/(6$/(6 352*5$0

2UGHUHGOLVW 3URJUDP1DPH &RPSXWDWLRQWLPHPV  FRPFD[[[\DDDWU[$$$B   FRPFD[[[\DDDWU[$$$B   FRPFD[[[\DD]WU[$$=B  7$%/(,9 7+($9(5$*((;(&87,212)6,1*/(6$/(6352*5$0 2UGHUHGOLVW 3URJUDP1DPH &RPSXWDWLRQWLPHPV

 FRPFD[[[\DDEWU[$$%B   FRPFD[[[\DD]WU[$$=B   FRPFD[[[\DDEWU[$$%B  [hadoop@~/pig_scripts]$./cust_analysis_main.sh Pleasechoose烉(1)Programcomputingcountstatics; (2)APserverflowstatics;==>2 Pleaseenterdates(FormatYYMMDD): 130304 Formoreinformation,pleaseenterthestartingtime (FormatHHMM): 1300 Formoreinformation,pleaseentertheendtime (FormatHHMM): 1700 PleaseenteraBusinessCode: AT

3URJUDP SHUIRUPDQFH DQDO\VLV SURYLGHG LQWHUDFWLYH LQSXW FRQGLWLRQVIHDWXUHV7KLVV\VWHPXVHVV\VWHPUHWULHYHVUHFRUGV WR DQDO\VLV WKH IXQFWLRQ UHVSRQVH VSHHG 7KH UHVXOW VKRZV DFWLRQ WRWDO FRPSXWLQJ WLPHV DQG VWDWLVWLFDO GDWD RQ WKH $3 6HUYHU 7KH GDWD DUH FRPSXWLQJ WLPHV RI DFWLRQ DYHUDJH FRPSXWLQJ WLPH WKH PRVW FRPSXWLQJ WLPH DQG WKH VKRUWHVW FRPSXWLQJWLPHUHVSHFWLYHO\7KHVSHFLILHGFRQGLWLRQVDQDO\VLV UHVXOWVRISURJUDPSHUIRUPDQFHDUHLQWKHIROORZ76%/(9 7$%/(9 7+(63(&,),('&21',7,216$1$/<6,65(68/762)352*5$0 3(5)250$1&( 4XHU\GDWH 4XHU\WLPHa 3URJUDP1DPHFRPFD[[[\DDHWU[$$(B 7KHWRWDOWLPHVVWDWLVWLFV $36HUYHU DS DS DS DS DS 6XEWRWDO &RPSXWLQJWLPHV       TXHU\DFWLRQVWDWLVWLFV $36HUYHU DS DS DS DS DS 6XEWRWDO $FWLRQFRPSXWLQJWLPHV       $YHUDJHFRPSXWLQJWLPH       0RVWFRPSXWLQJWLPH       6KRUWHVWFRPSXWLQJWLPH       E. Performance comparison ,QWKLVVHFWLRQZHIRFXVRQWKH+')6SHUIRUPDQFHDQGWHVW WZRSURJUDPVSHUIRUPDQFHRI3LJDQG6KHOOVFULSWV 7HVW  +')6 LV D GLVWULEXWHG ILOH V\VWHP 7KH GDWD LV FHQWUDOL]HG PDQDJHPHQW 1DPHQRGH DQG GLVWULEXWHG 6WRUDJH 'DWDQRGH +')6 ILOHV V\VWHP SDUWLWLRQ VL]H DQG FRS\ EORFN VL]HLV0%DQGVWRUHGLQ'DWDQRGH,WDIIHFWVWKHQXPEHURI 0DSLQ0DSUHGXFHDQGHIILFLHQF\ZKDWILOHVL]HDQGEORFNVL]H 7KH QXPEHU RI PDS UHODWLRQVKLS EHWZHHQ WKH ILOHV DQG EORFN VL]HLVIROORZLQJ

7RWDOGDWDVL]HEORFNVL]H PDSQXPEHU

:HDUHDFFRUGLQJWREORFNVL]HFXWWKHILOHVZKHQWKHILOHV RYHUEORFNVL]H$IWHUFXWWLQJZHGLVWULEXWHWRGLIIHUHQWQRGHIRU SURFHVVLQJ /DUJH EORFN VL]H OHW KDUG GLVN GDWD WUDQVIHU WLPH ORQJHU WKDQ VHDUFK IRU D ILOH VWDUWLQJ SRVLWLRQ VHHN WLPH 7KLV PDNHVWKHGDWDWUDQVIHUVSHHGRI+')6DQGKDUGGLVNWUDQVIHU VSHHG FORVHU 6PDOOHU EORFN VL]H OHW ELJ ILOHV FXWWLQJ WR PRUH VPDOOILOHV7KLVFDXVHDORWRIFRS\UHDGZULWHDQGPRUHGLVN VHHNWLPHWKHQGUDJRQ0DSSHUIRUPDQFH

:H WHVW WKH HIIHFW RQ SHUIRUPDQFH RI GLIIHUHQW EORFN VL]H DQGWU\WRVHDUFKWKHPRVWVXLWDEOHIRUEORFNVL]HRQWKLVV\VWHP 7HVW &RQGLWLRQV $ :HEORJ ILOHVDERXW 0% LV FXW 0%  0%  0% DQG 0% EORFN VL]H WKHQ XSORDGHG WR +')6DQGLPSOHPHQW6\VWHPWUDIILFWUHQGVEDWFKSURJUDP7KH WHVWUHVXOWLVLQWKH7$%/(9,

7$%/(9, %/2&.6,=(3(5)250$1&(&203$5,621 %ORFNVL]H &XWWLQJWKHQXPEHURIILOHV &RPSXWLQJWLPH

0%  PLQXWHVVHFRQGV 0%  PLQXWHVVHFRQGV 0%  PLQXWHVHFRQGV 0%  PLQXWHVHFRQGV  ,IWKHELJILOHVEORFNVL]HLVVRVPDOOWKDWFXWWLQJWRRPDQ\ ILOH DQG PHWDGDWD VWRUHG LQ PHPRU\ RI 1DPHQRGH FDQ¶W EH UHOHDVHGVRWDNHXSWRRPXFKUHVRXUFHV7KLVV\VWHP¶VZHEORJ VL]HLVDYHUDJHODUJHUWKDQ0%IRUDVLQJOHVDOHVILOHVRWKLV V\VWHPEORFNVL]HGHIDXOWVWR0% 7HVW/LQX[HQYLURQPHQWFDQXVHYHU\FRPPDQGVRUWRRO IRUSURFHVVLQJDQGDQDO\VLVWKHZRUGILOHV,QUHFHQW\HDUVILOHV DUHDOPRVWPRUHVHYHUDOKXQGUHG0%RU*%&RQWHQWDQDO\VLV UHTXLUHPHQWV DUH YDULDEOH WRR ,W LV YHU\ LPSUDFWLFDO DQG GLIILFXOWWRUHDFKXVHUVWKDWRQO\XVHVDPSOH/LQX[LQVWUXFWLRQ DQG6KHOOVFULSWV:HZURWHD6KHOOVFULSWVWRVLPXODWHSURJUDP FRPSXWLQJFRXQWVWDWLVWLFVIXQFWLRQ8VLQJWKLVIXQFWLRQWHVWHG GLIIHUHQWILOHVL]HDQGFRPSDUHZLWK+DGRRSDQGJHQHUDOZRUG SURFHVVLQJ WRROV SHUIRUPDQFH GLIIHUHQFHV 7KH UHVXOW LV LQ WKH IROORZLQJ)LJ



)LJ&RPSDUH3LJDQG6KHOOSURFHVVLQJWLPHPLQXWH

+DGRRS LV D GLVWULEXWHG DUFKLWHFWXUH DQG VXLWDEOH IRU KDQGOLQJ ODUJH ILOHV +DGRRS SHUIRUPDQFH LV EHWWHU WKDQ VKHOO FRPPDQG ZKHQ WKH ILOHV RYHU 0% 3LJ LV D KLJKOHYHO SURJUDPPLQJ ODQJXDJH ,W SURYLGHV VR ORWV RI GDWD DQDO\VLV IXQFWLRQV WKDW WR PHHW WKH GLIIHUHQW QHHGV 7KXV +DGRRS DUFKLWHFWXUH LV VXLWDEOH WR VROXWLRQ WKLV V\VWHP¶V DQDO\VLV UHTXLUHPHQWV

9 &21&/86,21

7KLV SDSHU SURSRVHV D :HEORJ DQDO\VLV V\VWHP EDVHG RQ +DGRRS7KHLPSOHPHQWDWLRQVKRZVWKDW0DS5HGXFHSURJUDP 6WUXFWXUHFDQHIIHFWLYHVROXWLRQYHU\ODUJH:HEORJILOHVLQWKH +DGRRSHQYLURQPHQW3LJSURJUDPPLQJODQJXDJHFDQEHHDVLO\ DQDO\]HGWKHUHTXLUHPHQWRIORJDQGEHWWHUSHUIRUPDQFH7KLV V\VWHP SURYLGHG WZR IXQFWLRQV IRU $SSOLFDWLRQ 6HUYHU WUDIILF DQDO\VLVDQGSURJUDPSHUIRUPDQFHVWDWLVWLFV$36HUYHUWUDIILF 6WDWLVWLFV SURYLGHG V\VWHP DGPLQLVWUDWRUV PRQLWRULQJ V\VWHP IRXQGWKHSRWHQWLDOSUREOHPVDQGSUHGLFWWKHIXWXUHWUHQGRIWKH V\VWHP3URJUDPSHUIRUPDQFHVWDWLVWLFVSURYLGHVDUHIHUHQFHIRU SHUIRUPDQFHWXQLQJ 5()(5(1&(6 >@ $SDFKH+DGRRS$YDLODEOHKWWSKDGRRSDSDFKHRUJ >@ $SDFKH1XWFK$YDLODEOHKWWSQXWFKDSDFKHRUJ >@ +DGRRS0DS5HGXFH$YDLODEOHKWWSKDGRRSDSDFKHRUJPDSUHGXFH >@ +DGRRS'LVWULEXWHG)LOH6\VWHP$YDLODEOH KWWSKDGRRSDSDFKHRUJKGIV >@ $SDFKH3LJ$YDLODEOHKWWSSLJDSDFKHSLJ >@ 3LJ/DWLQ$YDLODEOHKWWSZLNLDSDFKHRUJSLJ3LJ/DWLQ

>@ 1&+& &ORXG &RPSXWLQJ 5HVHDUFK *URXS $YDLODEOH

KWWSWUDFQFKFRUJWZFORXG

>@ +DGRRS 'RFXPHQWDWLRQ0DS5HGXFH 7XWRULDO 7KH $SDFKH 6RIWZDUH)RXQGDWLRQ

>@ ' %RUWKDNXU 7KH +DGRRS 'LVWULEXWHG )LOH 6\VWHP $UFKLWHFWXUH DQG 'HVLJQ7KH$SDFKH6RIWZDUH)RXQGDWLRQ

>@ -'HDQDQG6*KHPDZDW³0DS5HGXFHVLPSOLILHGGDWDSURFHVVLQJRQ ODUJH FOXVWHUV´ LQ 3URF RI WKH WK 86(1,; 6\PSRVLXP RQ 2SHUDWLQJ 6\VWHPV'HVLJQ ,PSOHPHQWDWLRQ26', >@ <<X0,VDUG')HWWHUO\0%XGLX8(UOLQJVVRQ3.*XQGDDQG -&XUUH\³'U\DG/,14$V\VWHPIRUJHQHUDOSXUSRVHGLVWULEXWHGGDWD SDUDOOHOFRPSXWLQJXVLQJDKLJKOHYHOODQJXDJH´ >@ &2OVWRQ%5HHG86ULYDVWDYD5.XPDUDQG$7RPNLQV³3LJODWLQ DQRWVRIRUHLJQODQJXDJHIRUGDWDSURFHVVLQJ´LQ3URFRIWKH6,*02' &RQISS± >@ 0DFNH\ * 6HKULVK 6 -XQ :DQJ ³,PSURYLQJ PHWDGDWD PDQDJHPHQW IRU VPDOO ILOHV LQ +')6´ ,((( ,QWHUQDWLRQDO &RQIHUHQFH RQ OXVWHU &RPSXWLQJDQG:RUNVKRSVSS±$XJ

>@ $*DWHV21DWNRYLFK6&KRSUD3.DPDWK61DUD\DQDP&2OVWRQ % 5HHG 6 6ULQLYDVDQ DQG 8 6ULYDVWDYD ³%XLOGLQJ D +LJK/HYHO 'DWDIORZ6\VWHPRQ7RSRI0DS5HGXFH7KH3LJ([SHULHQFH´3URFRI WKH9/'%(QGRZPHQWYROQR >@ 5&KDLNHQ%-HQNLQV3$/DUVRQ%5DPVH\'6KDNLE6:HDYHU DQG-=KRX³(DV\DQG(IILFLHQW3DUDOOHO3URFHVVLQJ RI0DVVLYH'DWD 6HWV´3URFRIWKH9/'%(QGRZPHQWYROQR