Optimal Allocation of Testing Resources for Modular Software Systems

2SWLPDO$OORFDWLRQRI7HVWLQJ5HVRXUFHVIRU0RGXODU6RIWZDUH6\VWHPV





&KLQ<X+XDQJ



_{-XQJ+XD/R}



_6\<HQ.XR



_{DQG0LFKDHO5/\X}



_



_{'HSDUWPHQWRI(OHFWULFDO(QJLQHHULQJ  }



_{&RPSXWHU6FLHQFH (QJLQHHULQJ'HSDUWPHQW}

        1DWLRQDO7DLZDQ8QLYHUVLW\            7KH&KLQHVH8QLYHUVLW\RI+RQJ.RQJ

7DLSHL7DLZDQ                          6KDWLQ+RQJ.RQJ

          V\N XR#FFHHQWXHGXWZ                     O\X#FVHFXKN HGXKN 





$EVWUDFW 

,Q WKLV SDSHU EDVHG RQ VRIWZDUH UHOLDELOLW\ JURZWK PRGHOV ZLWK JHQHUDOL]HG ORJLVWLF WHVWLQJHIIRUW IXQFWLRQ ZH VWXG\ WKUHH RSWLPDO UHVRXUFH DOORFDWLRQ SUREOHPV LQ PRGXODU VRIWZDUH V\VWHPV GXULQJ WHVWLQJ SKDVH  PLQLPL]DWLRQ RI WKH UHPDLQLQJ IDXOWV ZKHQ D IL[HG DPRXQW RI WHVWLQJHIIRUW DQG D GHVLUHG UHOLDELOLW\ REMHFWLYH DUH JLYHQ  PLQLPL]DWLRQ RI WKH UHTXLUHG DPRXQW RI WHVWLQJHIIRUW ZKHQ D VSHFLILF QXPEHU RI UHPDLQLQJ IDXOWV DQG D GHVLUHG UHOLDELOLW\ REMHFWLYH DUH JLYHQ DQG  PLQLPL]DWLRQ RI WKH FRVW ZKHQ WKH QXPEHU RI UHPDLQLQJ IDXOWV DQG D GHVLUHG UHOLDELOLW\ REMHFWLYH DUH JLYHQ 6HYHUDO XVHIXO RSWLPL]DWLRQ DOJRULWKPV EDVHG RQ WKH /DJUDQJH PXOWLSOLHU PHWKRG DUH SURSRVHG DQG QXPHULFDO H[DPSOHV DUH LOOXVWUDWHG 2XU PHWKRGRORJLHV SURYLGH SUDFWLFDO DSSURDFKHV WR WKH RSWLPL]DWLRQ RI WHVWLQJ UHVRXUFH DOORFDWLRQ ZLWK D UHOLDELOLW\ REMHFWLYH ,Q DGGLWLRQ ZH DOVR LQWURGXFH WKH WHVWLQJUHVRXUFH FRQWURO SUREOHP DQG FRPSDUH GLIIHUHQW UHVRXUFH DOORFDWLRQ PHWKRGV )LQDOO\ ZH GHPRQVWUDWH KRZ WKHVH DQDO\WLFDO DSSURDFKHV FDQ EH HPSOR\HG LQ WKH LQWHJUDWLRQ WHVWLQJ 8VLQJ WKH SURSRVHG DOJRULWKPV SURMHFW PDQDJHUV FDQ DOORFDWH OLPLWHG WHVWLQJ UHVRXUFH HDVLO\ DQG HIILFLHQWO\ DQG WKXV DFKLHYH WKH KLJKHVW UHOLDELOLW\ REMHFWLYH GXULQJ VRIWZDUH PRGXOH DQG LQWHJUDWLRQ WHVWLQJ



,QWURGXFWLRQ

0RGHUQ FRPSXWHU V\VWHPV FRQWDLQLQJ ERWK KDUGZDUH DQG VRIWZDUH KDYH EHFRPH YHU\ FRPSOH[ 6RIWZDUH WDNHV D ODUJH SRUWLRQ RI WKH V\VWHP FRVW DQG UHTXLUHV WKH DFKLHYHPHQW RI KLJK UHOLDELOLW\ LH D KLJK FRQILGHQFH LQ WKH DELOLW\ RI WKH VRIWZDUH WR SHUIRUP ZLWKRXW HUURU ,Q IDFW VRIWZDUH UHOLDELOLW\ UHSUHVHQWV D FXVWRPHURULHQWHG YLHZ RI VRIWZDUH TXDOLW\ ,W UHODWHV WR SUDFWLFDO RSHUDWLRQ RI D SURJUDP UDWKHU WKDQ VLPSO\ LWV GHVLJQ DQG LPSOHPHQWDWLRQ 0HDVXULQJ RU SUHGLFWLQJ VRIWZDUH UHOLDELOLW\ LV D FKDOOHQJH DQG LWV UHVXOW LV LPSRUWDQW DV D TXDQWLWDWLYH DVVHVVPHQW RI WKH SHUIRUPDQFH RI WKH XQGHUO\LQJ VRIWZDUH V\VWHPV 7KLV LV HVSHFLDOO\ WUXH EHIRUH WKH VRIWZDUH V\VWHPV DUH UHOHDVHG WR

WKH PDUNHW 7KHUHIRUH UHVHDUFK HIIRUWV LQ VRIWZDUH UHOLDELOLW\ HQJLQHHULQJ KDYH EHHQ FRQGXFWHG RYHU WKH SDVW WKUHH GHFDGHV DQG PDQ\ VRIWZDUH UHOLDELOLW\ JURZWK PRGHOV 65*0V KDYH EHHQ SURSRVHG >@ ,Q IDFW LQ DGGLWLRQ WR VRIWZDUH UHOLDELOLW\ PHDVXUHPHQW 65*0V FDQ DOVR KHOS XV WR SUHGLFW WKH IDXOW GHWHFWLRQ FRYHUDJH LQ WKH WHVWLQJ SKDVH

3UDFWLFDOO\ D VRIWZDUH WHVWLQJ SURFHVV FRQVLVWV RI VHYHUDO WHVWLQJ VWDJHV LQFOXGLQJ PRGXOH WHVWLQJ LQWHJUDWLRQ WHVWLQJ V\VWHP WHVWLQJ DQG LQVWDOODWLRQ WHVWLQJ 7KH TXDOLW\ RI WKH WHVWV XVXDOO\ FRUUHVSRQGV WR WKH PDWXULW\ RI WKH VRIWZDUH WHVW SURFHVV ZKLFK LQ WXUQ UHODWHV WR WKH PDWXULW\ RI WKH RYHUDOO VRIWZDUH GHYHORSPHQW SURFHVV >@ ,Q JHQHUDO PRVW SRSXODU DQG FRPPHUFLDO VRIWZDUH SURGXFWV DUH FRPSOH[ V\VWHPV DQG FRPSRVHG RI D QXPEHU RI PRGXOHV $V VRRQ DV WKH PRGXOHV DUH GHYHORSHG WKH\ KDYH WR EH WHVWHG LQ D YDULHW\ RI ZD\V ,I WKH PRGXOHV DUH VLPSOH HQRXJK WHVW FDVHV ZLWK DQDO\WLF VROXWLRQV FDQ EH JHQHUDWHG WR H[HUFLVH WKH PDWKHPDWLFDO DFFXUDF\ RI WKH PRGXOHV >@ 6LPLODU WHVWV FDQ EH UXQ ZLWK YDULRXV OHYHOV RI FRXSOLQJ RI WKH PRGXOHV )XUWKHUPRUH DOO WKH WHVWLQJ DFWLYLWLHV RI GLIIHUHQW PRGXOHV VKRXOG EH FRPSOHWHG ZLWKLQ D OLPLWHG WLPH VLQFH WKH\ ZLOO FRQVXPH DSSUR[LPDWHO\ a RI WKH WRWDO DPRXQW RI VRIWZDUH GHYHORSPHQW UHVRXUFHV >@ 7KHUHIRUH SURMHFW PDQDJHUV VKRXOG NQRZ KRZ WR DOORFDWH WKH VSHFLILHG WHVWLQJUHVRXUFHV DPRQJ DOO WKH PRGXOHV DQG GHYHORS TXDOLW\ VRIWZDUH ZLWK KLJK UHOLDELOLW\ 0DQ\ SDSHUV KDYH DGGUHVVHG WKH RSWLPDO UHVRXUFH DOORFDWLRQ SUREOHP RYHU WKH \HDUV LQFOXGLQJ .XEDW HW DO   >@ <DPDGD HW DO   >@ /HXQJ   >@ +RX HW DO  >@ ;LH HW DO   >@ DQG /\X HW DO  >@

,Q WKLV SDSHU ZH FRQVLGHU WKUHH NLQGV RI VRIWZDUH WHVWLQJUHVRXUFH DOORFDWLRQ SUREOHPV DQG SURSRVH VHYHUDO VWUDWHJLHV IRU PRGXOH WHVWLQJ LQ RUGHU WR KHOS WKH PDQDJHUV WR PDNH WKH EHVW GHFLVLRQV 7KDW LV ZH SURYLGH V\VWHPDWLF PHWKRGV IRU WKH VRIWZDUH SURMHFW PDQDJHUV WR DOORFDWH VSHFLILF DPRXQW RI WHVWLQJUHVRXUFH H[SHQGLWXUHV IRU HDFK PRGXOH XQGHU VRPH FRQVWUDLQWV VXFK DV  PLQLPL]LQJ WKH QXPEHU RI UHPDLQLQJ IDXOWV ZLWK D UHOLDELOLW\ REMHFWLYH  PLQLPL]LQJ WKH DPRXQW RI WHVWLQJHIIRUW ZLWK D UHOLDELOLW\ REMHFWLYH RU  PLQLPL]LQJ WKH FRVW ZLWK D UHOLDELOLW\

REMHFWLYH +HUH ZH XVH DQ 65*0 ZLWK JHQHUDOL]HG ORJLVWLF WHVWLQJHIIRUW IXQFWLRQ WR GHVFULEH WKH WLPHGHSHQGHQF\ EHKDYLRUV RI GHWHFWHG VRIWZDUH IDXOWV DQG WKH WHVWLQJ UHVRXUFH H[SHQGLWXUHV VSHQW GXULQJ PRGXOH WHVWLQJ %HVLGHV ZH DOVR VWXG\ D WHVWLQJUHVRXUFH DOORFDWLRQ VWUDWHJ\ IRU VRIWZDUH LQWHJUDWLRQ WHVWLQJ ZKHQ DOO PRGXOHV DUH LQWHUFRQQHFWHG DQG WHVWHG WRJHWKHU

7KH UHPDLQLQJ RI WKH SDSHU FRQVLVWV RI IRXU VHFWLRQV 6HFWLRQ  GHVFULEHV DQ 65*0 ZLWK JHQHUDOL]HG ORJLVWLF WHVWLQJHIIRUW IXQFWLRQ ZKLFK LV EDVHG RQ QRQKRPRJHQHRXV 3RLVVRQ SURFHVVHV 1+33V ,Q 6HFWLRQ  WKH PHWKRGV IRU WHVWLQJ UHVRXUFH DOORFDWLRQ DQG RSWLPL]DWLRQ IRU PRGXODU VRIWZDUH WHVWLQJ DUH LQWURGXFHG :H LQYHVWLJDWH WKHVH RSWLPL]DWLRQ SUREOHPV IRU WKUHH GLIIHUHQW UHTXLUHPHQWV PLQLPL]LQJ WKH QXPEHU RI UHPDLQLQJ IDXOWV PLQLPL]LQJ WKH DPRXQW RI WHVWLQJHIIRUW DQG PLQLPL]LQJ WKH FRVW 6HYHUDO QXPHULFDO H[DPSOHV IRU WKH RSWLPXP WHVWLQJHIIRUW DOORFDWLRQ SUREOHP DUH DOVR GHPRQVWUDWHG %HVLGHV D WHVWLQJ UHVRXUFHV FRQWURO SUREOHP DQG FRPSDULVRQV DPRQJ GLIIHUHQW UHVRXUFH DOORFDWLRQ PHWKRGV DUH SUHVHQWHG )XUWKHUPRUH WHVWLQJ UHVRXUFH DOORFDWLRQ DQG RSWLPL]DWLRQ IRU LQWHJUDWLRQ WHVWLQJ DUH GLVFXVVHG LQ 6HFWLRQV  )LQDOO\ WKH FRQFOXGLQJ UHPDUNV DUH JLYHQ LQ 6HFWLRQ 



 65*0 ZLWK  JHQHUDOL]HG ORJLVWLF WHVWLQJ

HIIRUWIXQFWLRQ



$ QXPEHU RI 65*0V KDYH EHHQ SURSRVHG RQ WKH VXEMHFW RI VRIWZDUH UHOLDELOLW\ $PRQJ WKHVH PRGHOV *RHO DQG 2NXPRWR XVHG DQ 1+33 DV WKH VWRFKDVWLF SURFHVV WR GHVFULEH WKH IDXOW SURFHVV >@ <DPDGD HW DO >@ PRGLILHG WKH *2 PRGHO DQG LQFRUSRUDWHG WKH FRQFHSW RI WHVWLQJHIIRUW LQ DQ 1+33 PRGHO WR JHW D EHWWHU GHVFULSWLRQ RI WKH VRIWZDUH IDXOW SKHQRPHQRQ /DWHU ZH DOVR SURSRVHG D QHZ 65*0 ZLWK WKH ORJLVWLF WHVWLQJHIIRUW IXQFWLRQ WR SUHGLFW WKH EHKDYLRU RI IDLOXUH RFFXUUHQFHV DQG WKH IDXOW FRQWHQW RI D VRIWZDUH SURGXFW %DVHG RXU SDVW H[SHULPHQWDO UHVXOWV WKLV SURSRVHG DSSURDFK LV ZHOO VXLWDEOH IRU HVWLPDWLQJ WKH UHOLDELOLW\ RI VRIWZDUH DSSOLFDWLRQ GXULQJ WKH GHYHORSPHQW SURFHVV >@ +HUH DUH WKH PRGHOLQJ DVVXPSWLRQV  7KH IDXOW UHPRYDO SURFHVV LV PRGHOHG E\ DQ 1+33  7KH VRIWZDUH DSSOLFDWLRQ LV VXEMHFW WR IDLOXUHV DW UDQGRP

WLPHV FDXVHG E\ WKH UHPDLQLQJ IDXOWV LQ WKH V\VWHP  7KH PHDQ QXPEHU RI IDXOWV GHWHFWHG LQ WKH WLPH LQWHUYDO

W W'W E\ WKH FXUUHQW WHVWLQJHIIRUW LV SURSRUWLRQDO WR WKH PHDQ QXPEHU RI UHPDLQLQJ IDXOWV LQ WKH V\VWHP DW WLPH W DQG WKH SURSRUWLRQDOLW\ LV D FRQVWDQW RYHU WLPH  7HVWLQJ HIIRUW H[SHQGLWXUHV DUH GHVFULEHG E\ D

JHQHUDOL]HG ORJLVWLF WHVWLQJHIIRUW IXQFWLRQ

 (DFK WLPH D IDLOXUH RFFXUV WKH FRUUHVSRQGLQJ IDXOW LV LPPHGLDWHO\ UHPRYHG DQG QR QHZ IDXOWV DUH LQWURGXFHG

%DVHG RQ WKH DVVXPSWLRQV LI WKH QXPEHU RI IDXOWV GHWHFWHG E\ WKH FXUUHQW WHVWLQJHIIRUW H[SHQGLWXUHV LV

SURSRUWLRQDO WR WKH QXPEHU RI UHPDLQLQJ IDXOWV WKHQ ZH REWDLQ WKH IROORZLQJ GLIIHUHQWLDO HTXDWLRQ

@  >      _{U D PW} W Z GW W GP _{u u } N



ZKHUH PW LV WKH H[SHFWHG PHDQ QXPEHU RI IDXOWV GHWHFWHG LQ WLPH  W  :NW LV WKH FXUUHQW WHVWLQJHIIRUW FRQVXPSWLRQ DW WLPH W D LV WKH H[SHFWHG QXPEHU RI LQLWLDO IDXOWV U LV WKH HUURU GHWHFWLRQ UDWH SHU XQLW WHVWLQJHIIRUW DW WHVWLQJ WLPH W DQG U! 6ROYLQJ WKH DERYH GLIIHUHQWLDO HTXDWLRQ XQGHU WKH ERXQGDU\ FRQGLWLRQ P  LH WKH PHDQ YDOXH IXQFWLRQ PW LV HTXDO WR ]HUR DW WLPH  ZH KDYH @      H[S>    W D U:_N W :_N P   



@   H[S>   U: W D  

           

 ,Q (T  PW LV QRQGHFUHDVLQJ ZLWK UHVSHFW WR WHVWLQJ WLPH W .QRZLQJ LWV YDOXH FDQ KHOS XV WR GHWHUPLQH ZKHWKHU WKH VRIWZDUH LV UHDG\ IRU UHOHDVH DQG KRZ PXFK PRUH WHVWLQJ UHVRXUFHV DUH UHTXLUHG >@ ,W FDQ SURYLGH DQ HVWLPDWH RI WKH QXPEHU RI IDLOXUHV WKDW ZLOO HYHQWXDOO\ EH HQFRXQWHUHG E\ WKH FXVWRPHUV %HVLGHV ZH DOVR SURSRVHG D JHQHUDOL]HG ORJLVWLF WHVWLQJHIIRUW IXQFWLRQ ZLWK VWUXFWXULQJ LQGH[ ZKLFK FDQ EH XVHG WR FRQVLGHU DQG HYDOXDWH WKH HIIHFWV RI SRVVLEOH LPSURYHPHQWV RQ VRIWZDUH GHYHORSPHQW PHWKRGRORJ\ VXFK DV WRSGRZQ GHVLJQ RU VWHSZLVH UHILQHPHQW >@ 7KH JHQHUDOL]HG ORJLVWLF WHVWLQJHIIRUW IXQFWLRQ LV GHSLFWHG DV IROORZ N N N _DNE          ¸¸¹ · ¨¨© §   u  W $H 1 W : 

ZKHUH 1 LV WKH WRWDO DPRXQW RI WHVWLQJ HIIRUW WR EH HYHQWXDOO\ FRQVXPHG D LV WKH FRQVXPSWLRQ UDWH RI WHVWLQJHIIRUW H[SHQGLWXUHV $ LV D FRQVWDQW DQG N LV D VWUXFWXULQJ LQGH[ ZLWK D ODUJH YDOXH IRU PRGHOLQJ ZHOOVWUXFWXUHG VRIWZDUH GHYHORSPHQW HIIRUWV ,Q WKLV FDVH WKH WHVWLQJ HIIRUW UHDFKHV LWV PD[LPXP YDOXH DW WLPH N DN N $ W PD[ OQ



7KH FRQGLWLRQDO UHOLDELOLW\ IXQFWLRQ DIWHU WKH ODVW IDLOXUH RFFXUV DW WLPH W LV REWDLQHG E\ @     H[S>  _   W 5 W W W PW W PW 5 { '  ' 



7DNLQJ WKH ORJDULWKP RQ ERWK VLGHV RI WKH DERYH HTXDWLRQ ZH REWDLQ        OQ5 W  P W'W P W



)URP WKH DERYH HTXDWLRQ DQG (T  ZH FDQ GHWHUPLQH WKH WHVWLQJ WLPH QHHGHG WR UHDFK D GHVLUHG UHOLDELOLW\ 5 :H

DOVR FDQ GHILQH DQRWKHU PHDVXUH RI VRIWZDUH UHOLDELOLW\ LH WKH UDWLR RI WKH FXPXODWLYH QXPEHU RI GHWHFWHG IDXOWV DW WLPH W WR WKH H[SHFWHG QXPEHU RI LQLWLDO IDXOWV >@

D W P W

5 {  





5W 5 1RWH WKDW 5W LV LQFUHDVLQJ IXQFWLRQ LQ W 8VLQJ

5W ZH FDQ HDVLO\ JHW WKH UHTXLUHG WHVWLQJ WLPH QHHGHG WR UHDFK WKH UHOLDELOLW\ REMHFWLYH 5 RU GHFLGH ZKHWKHU WKH

UHOLDELOLW\ REMHFWLYH FDQ EH UHDFKHG DW D VSHFLILHG WLPH ,I ZH NQRZ WKDW WKH VRIWZDUH UHOLDELOLW\ RI D FRPSXWHU V\VWHP KDV UHDFKHG DQ DFFHSWDEOH UHOLDELOLW\ OHYHO WKHQ ZH FDQ GHWHUPLQH WKH ULJKW WLPH WR UHOHDVH WKLV VRIWZDUH



 7HVWLQJUHVRXUFH DOORFDWLRQ IRU PRGXOH

WHVWLQJ

,Q WKLV VHFWLRQ ZH ZLOO FRQVLGHU VHYHUDO UHVRXUFH DOORFDWLRQ SUREOHPV EDVHG RQ DQ 65*0 ZLWK JHQHUDOL]HG ORJLVWLF WHVWLQJHIIRUW IXQFWLRQ GXULQJ VRIWZDUH WHVWLQJ SKDVH

$VVXPSWLRQV >@

 7KH VRIWZDUH V\VWHP LV FRPSRVHG RI 1 LQGHSHQGHQW PRGXOHV WKDW DUH WHVWHG LQGLYLGXDOO\ 7KH QXPEHU RI VRIWZDUH IDXOWV UHPDLQLQJ LQ HDFK PRGXOH FDQ EH HVWLPDWHG E\ DQ 65*0 ZLWK JHQHUDOL]HG ORJLVWLF WHVWLQJHIIRUW IXQFWLRQ

 )RU HDFK PRGXOH WKH IDLOXUH GDWD KDYH EHHQ FROOHFWHG DQG WKH SDUDPHWHUV RI HDFK PRGXOH FDQ EH HVWLPDWHG  7KH WRWDO DPRXQW RI WHVWLQJ UHVRXUFH H[SHQGLWXUHV

DYDLODEOH IRU WKH PRGXOH WHVWLQJ SURFHVVHV LV IL[HG DQG GHQRWHG E\ :

 ,I DQ\ RI WKH VRIWZDUH PRGXOHV LV IDXOW\ WKH ZKROH VRIWZDUH V\VWHP IDLOV

 7KH V\VWHP PDQDJHU KDV WR DOORFDWH WKH WRWDO WHVWLQJ UHVRXUFHV : WR HDFK VRIWZDUH PRGXOH WR PLQLPL]H WKH QXPEHU RI IDXOWV UHPDLQLQJ LQ WKH V\VWHP GXULQJ WKH WHVWLQJ SHULRG %HVLGHV WKH GHVLUHG VRIWZDUH UHOLDELOLW\ DIWHU WKH WHVWLQJ SKDVH LV JUHDWHU WKDQ RU HTXDO WR WKH UHOLDELOLW\ REMHFWLYH 5

)URP 6HFWLRQ  WKH PHDQ YDOXH IXQFWLRQ RI D VRIWZDUH V\VWHP ZLWK 1 PRGXOHV FDQ EH IRUPXODWHG DV

0W ¦ ¦1  

L L L L L

1

LYLPLW YD H[S U: W 

ZKHUH YL LV D ZHLJKWLQJ IDFWRU WR PHDVXUH WKH UHODWLYH

LPSRUWDQFH RI D IDXOW UHPRYDO IURP PRGXOH L LQ WKH IXWXUH ,I YL  IRU DOO L  « 1 WKH REMHFWLYH LV WR PLQLPL]H WKH

WRWDO QXPEHU RI IDXOWV UHPDLQLQJ LQ WKH VRIWZDUH V\VWHP DIWHU WKLV WHVWLQJ SKDVH >@ 7KLV LQGLFDWHV WKDW WKH QXPEHU RI UHPDLQLQJ IDXOWV LQ WKH V\VWHP FDQ EH HVWLPDWHG E\ ¦  { ¦  1 L L L L L 1 LYLDLH[S UL:LW YD H[S U: 

:H FDQ IXUWKHU IRUPXODWH WKUHH RSWLPL]DWLRQ SUREOHPV DV IROORZV



 0LQLPL]LQJ WKH QXPEHU RI UHPDLQLQJ IDXOWV ZLWK

D JLYHQ IL[HG DPRXQW RI WHVWLQJHIIRUW DQG D UHOLDELOLW\

REMHFWLYH

)LUVWO\ WKH RSWLPL]DWLRQ SUREOHP LV WKDW WKH WRWDO DPRXQW RI WHVWLQJHIIRUW LV IL[HG DQG ZH ZDQW WR DOORFDWH WKHVH HIIRUWV WR HDFK PRGXOH LQ RUGHU WR PLQLPL]H WKH QXPEHU RI UHPDLQLQJ IDXOWV LQ WKH VRIWZDUH V\VWHPV 6SHFLDOO\ VXSSRVH WKH WRWDO DPRXQW RI WHVWLQJHIIRUW LV : DQG PRGXOH L LV DOORFDWHG :L WHVWLQJHIIRUW DQG WKHQ WKH

RSWLPL]DWLRQ SUREOHP FDQ EH UHSUHVHQWHG DV IROORZV 7KH REMHFWLYH IXQFWLRQ LV 0LQLPL]H _¦1 _ LYLDLH[S UL:L  6XEMHFW WR WKH FRQVWUDLQV    d t ¦ L 1 L :L : :  L   1     H[S  U: W 5 5  L L t 



)URP (T  ZH FDQ REWDLQ :L        OQ   L 1 5 U : L L t   /HW _{OQ 5 L 1} U ' L L {  7KXV ZH FDQ KDYH 1 L : : : L 1 L¦ L d  t  DQG : t L &L ZKHUH &L PD[''''1



7KDW LV WKH RSWLPDO WHVWLQJ UHVRXUFH DOORFDWLRQ FDQ EH GHSLFWHG DV EHORZ >@ 0LQLPL]H ¦1  LYLDLH[S UL:L  6XEMHFW WR _{ }  t

¦

L 1 L :L : : DQG : t_L &_L ZKHUH &L PD[''''1

/HW ;_L :_L&_L ZH FDQ WUDQVIRUP WKH DERYH HTXDWLRQV WR

0LQLPL]H



_¦1 _{ } LYLDLH[S UL&LH[S UL;L



 6XEMHFW WR _{ }   d ¦ t ¦ L 1 L L 1 L ;L : & ;  L   1 

1RWH WKDW WKH SDUDPHWHUV YL DL DQG UL VKRXOG DOUHDG\ EH

HVWLPDWHG E\ WKH SURSRVHG PRGHO 7R VROYH WKH DERYH SUREOHP WKH /DJUDQJH PXOWLSOLHU PHWKRG FDQ EH DSSOLHG &RQVHTXHQWO\ (T  DQG (T  FDQ EH VLPSOLILHG DV IROORZV

0LQLPL]H ¦1   L L L L L L L 1 YD U& U; ; ; ; /       H[S H[S   O      ¦ ¦  1 L L 1 L ;L : & O  %DVHG RQ WKH .XKQ7XFNHU FRQGLWLRQV >@ WKH QHFHVVDU\ FRQGLWLRQV IRU D PLQLPXP YDOXH RI (T  WR H[LVW DUH DV IROORZV $      _ w w L 1 ; ; ; ; / O _{ L   1} $ O t $ _{ }   d ¦ t ¦ L 1 L L 1 L ;L : & :  L   1

7KHUHIRUH IURP (T  ZH KDYH

u  u  w w  H[S        L L L L L L 1 _{YD U U&} ; ; ; ; / O   H[SUL;L O            _{¦  ¦} w w 1 L L 1 L L 1 _{; : &} ; ; ; / O O

_

7KXV WKH VROXWLRQ  L ; LV L L L L L L L YDU U& U ; OQ _uH[S_{ }OQ_O  L   1  7KH VROXWLRQ

O

 LV » » » ¼ º « « « ¬ ª ¦ ¦ u   ¦ 1 L L 1 L 1 L L L L L L L L U & : & U U D Y U          H[S OQ    H[S O



 +HQFH ZH JHW            1 ; ; ; ; ; DV DQ RSWLPDO

VROXWLRQ WR (T  +RZHYHU WKH DERYH ; _{PD\ KDYH}

VRPH QHJDWLYH FRPSRQHQWV LI _{uH[S O} L L L L LDU U& Y  PDNLQJ _{; LQIHDVLEOH IRU (T  DQG (T  ,I WKLV LV}

WKH FDVH WKH VROXWLRQ ;_{FDQ EH FRUUHFWHG E\ WKH IROORZLQJ}

VWHSV



$OJRULWKP  6WHS  6HW O  6WHS  &DOFXODWH WKH IROORZLQJ HTXDWLRQV @ OQ  H[S >OQ  _{u   O} L L L L L L L _U YDU U& ;  L   1  O » » » ¼ º « « « ¬ ª ¦ ¦ u   ¦ 1 L L O 1 L 1 L L L L L L L U & : & U LU D Y U         H[S OQ    H[S O 6WHS  5HDUUDQJH WKH LQGH[ L VXFK WKDW     ; ;1 O ; t t t  6WHS  ,) _t O 1 ; WKHQ VWRS LH WKH VROXWLRQ LV RSWLPDO (OVH  O 1 ;  O O (QG,) 6WHS  *R WR 6WHS 

7KH RSWLPDO VROXWLRQ KDV WKH IROORZLQJ IRUP

° ° ° ¯ ° ° ° ® ­ » » » ¼ º « « « ¬ ª ¦ ¦ u   ¦    u   RWKHUZLVH ; U & : & U U D Y U ZKHUH O 1 L U & U U D Y ; L O 1 L L O 1 L 1 L L L L L L L L L L L L L L L        H[S OQ    H[S        OQ  H[S  OQ    O O







$OJRULWKP  DOZD\V FRQYHUJHV LQ DW ZRUVW 1 VWHSV 7KXV WKH YDOXH RI REMHFWLYH IXQFWLRQ JLYHQ E\ (T  DW WKH RSWLPDO VROXWLRQ       ; ;1 ; LV ¦1   L YLDL UL&L UL;L  H[S  H[S 



 0LQLPL]LQJ WKH DPRXQW RI WHVWLQJHIIRUW JLYHQ WKH

QXPEHU RI UHPDLQLQJ IDXOWV DQG D UHOLDELOLW\ REMHFWLYH

6HFRQGO\ VXSSRVH = VSHFLILHV WKH QXPEHU RI UHPDLQLQJ IDXOWV LQ WKH V\VWHP DQG ZH KDYH WR DOORFDWH DQ DPRXQW RI WHVWLQJHIIRUW WR HDFK VRIWZDUH PRGXOH WR PLQLPL]H WKH WRWDO WHVWLQJHIIRUW 7KH RSWLPL]DWLRQ SUREOHP FDQ EH UHSUHVHQWHG DV IROORZV 7KH REMHFWLYH IXQFWLRQ LV 0LQLPL]H _¦1 L:L  6XEMHFW WR WKH FRQVWUDLQV     H[S  t ¦  L 1 L YLDL UL:L = :     H[S  U: W 5 5  L L t 



6LPLODUO\ IURP (T  ZH FDQ REWDLQ :L        OQ   L 1 5 U : L Lt  

)ROORZLQJ WKH VLPLODU VWHSV GHVFULEHG LQ VHFWLRQ  DQG OHW

L L L : & ;   ZH FDQ WUDQVIRUP DERYH HTXDWLRQV WR 0LQLPL]H _¦1  L;L &L  6XEMHFW WR    H[S  H[S   t ¦YD U& U; = ;L &L 1 L L L L L L L 

7R VROYH WKLV SUREOHP WKH /DJUDQJH PXOWLSOLHU PHWKRG FDQ EH XVHG (T  DQG (T  DUH FRPELQHG DV WKH IROORZLQJ HTXDWLRQ 0LQLPL]H _¦1 _{ } L 1 ;L &L ; ; ; /          O



¦1    L YLDLH[S UL&LH[S UL;L =  O

 





)URP WKH .XKQ7XFNHU FRQGLWLRQV WKH QHFHVVDU\ FRQGLWLRQV IRU D PLQLPXP RI (T  WR H[LVW DUH  H[S        L L L L L L 1 _{YD U U&} ; ; ; ; / _{ } w w O _O

_

   H[SUL;L     H[S  H[S         _{¦   } w w 1 L L L L L L L 1 _{YD U& U; =} ; ; ; / O O

                                     

 7KXV WKH VROXWLRQ  L ; LV



L L L L L LDU U& U Y ;_L OQ>_O H[S_ @ _{ L   1 } 7KH VROXWLRQO LV = U 1 L¦  L  O 



7KDW LV L L L L L L _U U & U = U D Y ; 1 L _L L OQ H[S ¦_  L   1  +HQFH ZH JHW



 



      _{  } 1 ; ; ; ; ; DV DQ RSWLPDO

VROXWLRQ WR (T  +RZHYHU WKH DERYH ; _{PD\ KDYH}

VRPH QHJDWLYH FRPSRQHQWV LI ¦   ₁ L U_L = & U U D YL LL L L    H[S  PDNLQJ ;_{LQIHDVLEOH IRU (T   ,Q WKLV FDVH WKH}

VROXWLRQ ; _{FDQ EH FRUUHFWHG E\ WKH IROORZLQJ VWHSV}

6LPLODUO\ ZH SURSRVH D VLPSOH DOJRULWKP WR GHWHUPLQH WKH RSWLPDO VROXWLRQ IRU WKH WHVWLQJHIIRUW DOORFDWLRQ SUREOHP $OJRULWKP  6WHS  6HW O  6WHS  &DOFXODWH @    H[S >OQ   ¦  1O L _L L L _U Y₌D U U& _U ; L L L L L  L   1 O 6WHS  5HDUUDQJH WKH LQGH[ L VXFK WKDW     ; ;1 O ; t t t  6WHS  ,) _t O 1 ; WKHQ VWRS (OVH XSGDWH  O 1 ;  O O (QG ,) 6WHS  *R WR 6WHS 

7KH RSWLPDO VROXWLRQ KDV WKH IROORZLQJ IRUP

     @    H[S >OQ   _{L 1 O} U & U = U D Y U ; 1 O L _L L L L LL _{L L} ¦  

$OJRULWKP  DOZD\V FRQYHUJHV LQ DW ZRUVW 1  VWHSV

 0LQLPL]LQJ WKH FRVW JLYHQ WKH QXPEHU RI

UHPDLQLQJ IDXOWV DQG D UHOLDELOLW\ REMHFWLYH

7KLUGO\ VXSSRVH WKH QXPEHU RI UHPDLQLQJ IDXOWV LQ WKH V\VWHP LV VWLOO VSHFLILHG E\ = :H VKRXOG DOORFDWH DQ DPRXQW RI WHVWLQJHIIRUW WR HDFK VRIWZDUH PRGXOH WR PLQLPL]H WKH FRVW RI IXUWKHU VRIWZDUH GHYHORSPHQW DQG WHVW 7KHUHIRUH WKH RSWLPL]DWLRQ SUREOHP FDQ EH UHSUHVHQWHG DV IROORZV 7KH REMHFWLYH IXQFWLRQ LV 0LQLPL]H    ¦1 L &RVWL :L   6XEMHFW WR WKH FRQVWUDLQV    H[S  t d 

¦

L 1 L L L L L : = : U D Y      H[S  U: W 5 5  L L t 



)URP (T  ZH FDQ REWDLQ :L        OQ   L 1 5 U : L Lt  

6LPLODUO\ XVLQJ WKH PHWKRGV GHVFULEHG LQ VHFWLRQ  DQG OHW

L L L : & ;   ZH FDQ WUDQVIRUP WKH DERYH HTXDWLRQV WR 0LQLPL]H    ¦1  L &RVWL ;L &L   6XEMHFW WR _{H[S H[S    }  t ¦   L 1 L YLDL UL&L UL;L = ; 

+HUH ZH DVVXPH WKDW WKH FRVW IXQFWLRQ &RVWL:L LV WKH FRVW

UHTXLUHG WR GHYHORS PRGXOH L ZLWK :LDQG LV GLIIHUHQWLDEOH >@

6LPLODUO\ XVLQJ WKH /DJUDQJH PXOWLSOLHU PHWKRG WKH DERYH HTXDWLRQ FDQ EH VLPSOLILHG DV IROORZV 0LQLPL]H   H[S H[S    

&RVW ; & YD U& U; =

/ 1 L L L L L L L 1 L¦ L L L O ¦   

                                     

 $JDLQ EDVHG RQ WKH .XKQ7XFNHU FRQGLWLRQV >@ WKH QHFHVVDU\ FRQGLWLRQV IRU REWDLQLQJ WKH PLQLPXP RI (T  DUH



u  u  w  w w w   _H[S  L L L L L L L L & U U D Y ; & ; &RVW ; / L L _O   H[SUL;L



L   1 

  H[S  H[S  ¦    w w 1 L YLDL UL&L UL;L = / O 



+HUH OHW L L ; & ; &RVW ; L L L L  w _w   ]  WKHQ (T  EHFRPHV   H[S  H[S    u  u  w w L L L L L L L L L L ; U & U U D Y ; / ; ] O  7KH RSWLPDO VROXWLRQ  L ; LV   H[S  H[S   _{ u }  L L L LL L L L L _{YDU U&} ; U ; _O ] _

,Q IDFW WKH DERYH LV D VLPSOH RSWLPL]DWLRQ SUREOHP DQG FDQ EH HDVLO\ VROYHG E\ QXPHULFDO PHWKRGV >@

 1XPHULFDO H[DPSOHV

,Q WKLV VXEVHFWLRQ WKUHH QXPHULFDO H[DPSOHV IRU WKH RSWLPXP WHVWLQJHIIRUW DOORFDWLRQ SUREOHP DUH GHPRQVWUDWHG +HUH ZH DVVXPH WKDW WKH HVWLPDWHG SDUDPHWHUV DL UL DQG N LQ

(T  IRU L  «  DUH VXPPDUL]HG LQ 7DEOH  0RUHRYHU WKH ZHLJKWLQJ YHFWRUV YL LQ (T  DUH DOVR

OLVWHG,Q WKLV VXEVHFWLRQ WKUHH QXPHULFDO H[DPSOHV IRU WKH RSWLPXP WHVWLQJHIIRUW DOORFDWLRQ SUREOHP DUH GHPRQVWUDWHG +HUH ZH DVVXPH WKDW WKH HVWLPDWHG SDUDPHWHUV DL UL DQG N LQ

(T  IRU L  «  DUH VXPPDUL]HG LQ 7DEOH  0RUHRYHU WKH ZHLJKWLQJ YHFWRUV YLLQ (T  DUH DOVR OLVWHG

,Q WKH IROORZLQJ ZH LOOXVWUDWH WKUHH H[DPSOHV WR VKRZ KRZ WKH RSWLPDO DOORFDWLRQ RI WHVWLQJHIIRUW H[SHQGLWXUHV WR HDFK VRIWZDUH PRGXOH LV GHWHUPLQHG

 ([DPSOH RI PLQLPL]LQJ WKH QXPEHU RI UHPDLQLQJ IDXOWV JLYHQ D IL[HG DPRXQW RI WHVWLQJHIIRUW DQG D UHOLDELOLW\ UHTXLUHPHQW 6XSSRVH WKH WRWDO DPRXQW RI WHVWLQJHIIRUW H[SHQGLWXUHV : LV JLYHQ %HVLGHV DOO WKH SDUDPHWHUV DL DQG UL RI (T  IRU HDFK VRIWZDUH PRGXOH

KDYH EHHQ HVWLPDWHG E\ XVLQJ WKH PD[LPXP OLNHOLKRRG HVWLPDWLRQ 0/( RU WKH OHDVW VTXDUHV HVWLPDWLRQ /6( :H DSSO\ WKH SURSRVHG PRGHO WR DFWXDO VRIWZDUH IDLOXUH GDWD >@ :H KDYH WR DOORFDWH WKH H[SHQGLWXUHV WR HDFK PRGXOH WR PLQLPL]H WKH QXPEHU RI UHPDLQLQJ IDXOWV )URP 7DEOH  DQG $OJRULWKP  LQ 6HFWLRQ  WKH RSWLPDO WHVWLQJHIIRUW H[SHQGLWXUHV IRU WKH VRIWZDUH V\VWHPV DUH HVWLPDWHG DQG VKRZQ LQ 7DEOH  )XUWKHUPRUH WKH QXPEHUV RI LQLWLDO IDXOWV WKH HVWLPDWHG UHPDLQLQJ IDXOWV RI WKH WKUHH H[DPSOHV DQG WKH UHGXFWLRQ LQ WKH QXPEHU RI UHPDLQLQJ IDXOWV DUH DOVR VKRZQ LQ 7DEOH 

Table 1: The estimated values of ai, ri, vi , and N

0RGXOH DL UL N YLLQ H[ YLLQ H[  YLLQ H[                                                                       

Table 2: The optimal testing-effort expenditures using algorithm 1. 0RGXOH L L

&

; 

IRU H[ L L

&

; 

IRU H[ L L

&

; 

IRU H[                                        

Table 3: The reduction in the number of remaining faults.

([DPSOHV ,QLWLDO IDXOWV 5HPDLQLQJ

IDXOWV 5HGXFWLRQ 

   

   

   

 ([DPSOH RI PLQLPL]LQJ WKH DPRXQW RI WHVWLQJHIIRUW JLYHQ WKH QXPEHU RI UHPDLQLQJ IDXOWV DQG UHOLDELOLW\ UHTXLUHPHQWV 6XSSRVH WKH WRWDO QXPEHU RI UHPDLQLQJ IDXOWV = LV JLYHQ :H KDYH WR DOORFDWH WKH H[SHQGLWXUHV WR HDFK PRGXOH WR PLQLPL]H WKH WRWDO DPRXQW RI WHVWLQJHIIRUW H[SHQGLWXUHV 8VLQJ $OJRULWKP  LQ 6HFWLRQ  DQG 7DEOH  WKH RSWLPDO VROXWLRQV DUH GHULYHG DQG VKRZQ LQ 7DEOH  )XUWKHUPRUH WKH UHODWLRQVKLS EHWZHHQ WKH WRWDO DPRXQW RI WHVWLQJHIIRUW H[SHQGLWXUHV DQG WKH UHGXFWLRQ UDWH RI WKH UHPDLQLQJ IDXOWV DUH DOVR GHSLFWHG LQ )LJXUH 

Table 4: The optimal testing-effort expenditures using algorithm 2. 0RGXOH L L

&

; 

IRU H[ L L

&

; 

IRU H[ L L

&

; 

IRU H[                                        

Figure 1: The reduction rate of remaining faults vs. the total testing-effort expenditures.

 7HVWLQJ UHVRXUFHV FRQWURO SUREOHP

,Q RUGHU WR UHGXFH WKH ULVN DQG DFKLHYH D JLYHQ RSHUDWLRQDO TXDOLW\ DW D VSHFLILHG WLPH ZH FDQ XVH 650*V WR HVWLPDWH DQG FRQWURO WKH UHTXLUHG WHVWLQJ HIIRUW 7KH PDLQ SUREOHP LV KRZ WR HVWLPDWH WKH QXPEHU RI H[WUD IDXOWV GXULQJ PRGXOH WHVWLQJ ZKLFK KDYH WR EH IRXQG >  @ /HW XV FRQVLGHU WKH IROORZLQJ VFHQDULR

 'XH WR HFRQRPLF FRQVLGHUDWLRQV VRIWZDUH WHVWLQJ DQG GHEXJJLQJ ZLOO HYHQWXDOO\ KDYH WR VWRS DW D WLPH SRLQW 7

 %DVHG RQ WKH 65*0 VHOHFWHG E\ WKH VRIWZDUH GHYHORSHUV RU WHVW WHDPV IRU HDFK PRGXOH LWV EHKDYLRU RI FRQVXPLQJ WHVWLQJ UHVRXUFHV GXULQJ WHVWLQJ LV HVWLPDWHG DW WLPH 777 7KHUHIRUH WKH SURMHFW PDQDJHU

KDV WR GHFLGH KRZ WR VSHQG : LZKHUH : tL &L DQG `      PD[^ ' ' ' '1 &L %DVHG RQ WKH DERYH VFHQDULR LI ZH NQRZ KRZ WR DGMXVW WKH FRQVXPSWLRQ UDWH RI WHVWLQJ HIIRUW H[SHQGLWXUHV LQ WKH ORJLVWLF WHVWLQJ HIIRUW IXQFWLRQ LH DL DW 7 WKHQ WKH SURMHFW

PDQDJHU FDQ LQFUHDVH WKH WHVWLQJUHVRXUFH H[SHQGLWXUHV LQ77@ )LUVW IURP (T  ZH NQRZ WKDW WHVWLQJHIIRUW FRQVXPSWLRQ IXQFWLRQ DW 7IRU HDFK PRGXOH LV DV IROORZV

 

           N N N D N N D N D N  ¸ ¸ ¹ · ¨ ¨ © §   u  _ W H $ W H $ 1 W Z L L L L L L L  7KHUHIRUH WKH PHDQ WHVWLQJHIIRUW FRQVXPSWLRQ IXQFWLRQ LQ

@  7 7 LV

 

           N N N D N N D N D N  ¸ ¸ ¹ · ¨ ¨ © §   u  _ W H $ W H $ 1 W Z L L L L L L L  DQG

 

                 N N N D N D N D D N D D       u  

¸

¸

¸

¹

·

¨

¨

¨

©

§

W H $ W H $ W H $ W H $ 1 1 L L L L L L L L L L L L W  ZKHUH 7_ tW!7_ $SSO\LQJ (T  ZH NQRZ

³

³

           7 L L 7 7ZLN W GW : Z N W GW  7KDW LV  ¸¸ ¸ ¸ ¸ ¹ · ¨¨ ¨ ¨ ¨ © §    ¸¸ ¸ ¸ ¸ ¹ · ¨¨ ¨ ¨ ¨ © §   N D N N D N           $ H 7 1 7 H $ 1 : L L L L L L L ¸¸ ¸ ¸ ¸ ¹ · ¨¨ ¨ ¨ ¨ © §    N DN     $ H 7 1 L L L





7KHUHIRUH DLFDQ EH HDVLO\ VROYHG E\ QXPHULFDO PHWKRGV IRU

WKH VHOHFWHG 65*0 7KH WHVW WHDPV FDQ JHW WKH PRGLILHG WHVWLQJHIIRUW FRQVXPSWLRQ IXQFWLRQ LQ 77@DQG SUHGLFW WKH FXPXODWLYH QXPEHU RI IDXOWV DW WLPH 7 GXULQJ PRGXOH

LQWHUYDO77@

 &RPSDULVRQV EHWZHHQ GLIIHUHQW UHVRXUFH

DOORFDWLRQ PHWKRGV

,Q JHQHUDO D WHVWLQJHIIRUW DOORFDWLRQ PHWKRG LV FDOOHG DQ DYHUDJH DOORFDWLRQ PHWKRG LI WKH WRWDO DPRXQW RI WHVWLQJHIIRUW LV DOORFDWHG WR HDFK PRGXOH HYHQO\ 7KDW LV

1 & : ; 1 L L DYH L ¦   _{IRU PRGXOH L   1 }

/HW =DYH EH WKH QXPEHU RI UHPDLQLQJ IDXOWV LQ WKH ZKROH

V\VWHP E\ WKH DYHUDJH WHVWLQJHIIRUW DOORFDWLRQ PHWKRG WKHQ ZH KDYH ¦1   L L DYH L L L L L DYH YD U& U; =  H[S H[S  

6LPLODUO\ D WHVWLQJHIIRUW DOORFDWLRQ PHWKRG LV FDOOHG D SURSRUWLRQDO DOORFDWLRQ PHWKRG LI WKH WRWDO DPRXQW RI WHVWLQJHIIRUW DOORFDWHG WR PRGXOH L LV SURSRUWLRQDO WR WKH QXPEHU RI UHPDLQLQJ IDXOWV LQ PRGXOH L 7KDW LV

¦  ¦     u 1 L L 1 L L L L L L L L L L L SURS L & ; U & U D ; U & U D : ;   H[S H[S   H[S  H[S _ L   1  /HW =SURS EH WKH QXPEHU RI UHPDLQLQJ IDXOWV LQ WKH ZKROH

V\VWHP E\ WKH SURSRUWLRQDO WHVWLQJHIIRUW DOORFDWLRQ PHWKRG WKHQ ZH KDYH ¦1   L SURS L L L L L LD U& U; Y =SURS  H[S H[S  

7KHUHIRUH ZLWK =DYH =SURS DQG =RSW ZH FDQ NQRZ ZKHWKHU

WKH RSWLPDO DOORFDWLRQ PHWKRG LV EHWWHU WKDQ WKH DYHUDJH RU SURSRUWLRQDO DOORFDWLRQ PHWKRGV RU QRW

 7HVWLQJUHVRXUFH DOORFDWLRQ IRU LQWHJUDWLRQ WHVWLQJ

,QWHJUDWLRQ WHVWLQJ LQYROYHV WHVWLQJ WKH FRPELQDWLRQV RI SURJUDP PRGXOHV DQG WKHLU LQWHUIDFHV ,QWHJUDWLRQ WHVWLQJ VKRXOG EH ZHOO SODQQHG GXULQJ WKH GHVLJQ SKDVH DQG WKHQ DFFRPSOLVKHG ZLWK DQ DSSURSULDWH EDODQFH RI GHYHORSHUV ZLWK GHVLJQ NQRZOHGJH DQG LQGHSHQGHQW WHVWHUV ZLWK PLQLPDO GHVLJQ ELDVHV 'XULQJ WKLV WHVWLQJ SKDVH D IL[HG DPRXQW RI WHVW FDVHV LV SUHSDUHG UHSUHVHQWLQJ DFWXDO YDOXHV IURP SURJUDP¶V LQSXW GRPDLQ +HUH ZH DVVXPH WKDW 1 PRGXOHV FRPSULVH WKH ZKROH VRIWZDUH V\VWHP DQG WKH VHOHFWHG WHVW FDVHV FDQ EH GLYLGHG LQWR 6 FDWHJRULHV GXULQJ WKH LQWHJUDWLRQ WHVWLQJ /HW DLN EH WKH WRWDO WKURXJKSXW RI

WHVW FDVHV RQ PRGXOH L IRU WHVW GDWD LQ FDWHJRU\ N $V ZH NQRZ PRUH H[HFXWLRQV RI D VRIWZDUH PRGXOH ZLOO LQFUHDVH WKH FKDQFHV RI ILQGLQJ WKH UHPDLQLQJ XQGHWHFWHG IDXOWV LQ WKDW PRGXOH ,Q JHQHUDO LI WKH WRWDO WKURXJKSXW RI WHVW GDWD LV ODUJH WKHQ WKH PHDQ GHWHFWLRQ WLPH RI D IDXOW ZLOO EH GHFUHDVHG 7KXV WKH IDXOW GHWHFWLRQ UDWH RI PRGXOH L FDQ EH

GHQRWHG DV ULDLN DQG ZH FDQ DVVXPH WKDW ULDLN LV DQ

LQFUHDVLQJ IXQFWLRQ RI WKURXJKSXW > @ +HUH GXH WR WKH OLPLWDWLRQ RI VSDFH ZH RQO\ XVH WKH UHVRXUFH DOORFDWLRQ SUREOHP GHVFULEHG LQ 6HFWLRQ  IRU LOOXVWUDWLRQ 6LPLODU DQDO\VLV DQG GLVFXVVLRQ FDQ EH DSSOLHG WR RWKHU FDVHV LQ 6HFWLRQ  DQG  7KH SUREOHP GHVFULEHG LQ 6HFWLRQ  PLQLPL]LQJ WKH QXPEHU RI UHPDLQLQJ IDXOWV JLYHQ D IL[HG DPRXQW RI WHVWLQJHIIRUW DQG D UHOLDELOLW\ UHTXLUHPHQW FDQ EH IRUPXODWHG DV 7KH REMHFWLYH IXQFWLRQ LV 0LQLPL]H _{¦ ¦}6 _{ u} N 1 L L L L LN N : U D Y      H[S D  6XEMHFW WR WKH FRQVWUDLQV    t ¦ N 6 N :N : :  N   6      H[S  U : 5 5   L DLN N t 

)URP (T  ZH FDQ REWDLQ _OQ 

    5 U : LN L N t _D  

)ROORZLQJ WKH VLPLODU VWHSV GHVFULEHG LQ VHFWLRQ  DQG OHW

L L L : & ;   ZH FDQ WUDQVIRUP WKH DERYH HTXDWLRQV WR 0LQLPL]H _{¦ ¦}6 _{ } N 1 L L L L LN N L LN N ; U & U D Y      H[S    H[S D D  6XEMHFW WR _{ }   d ¦ t ¦ L L N ; : & ; 6 N 6 N  N   6 

6LPLODUO\ XVLQJ WKH /DJUDQJH PXOWLSOLHU PHWKRG WKH DERYH HTXDWLRQ FDQ EH VLPSOLILHG DV IROORZV



¦ ¦6  u N 1 L L L L LN N 6 YD U & : : : /        H[S     O D



     H[S   ¦   ¦   1 L L 1 L L & : ; ; UL DLN N O



u ¦   w w 1 L L L L LN L LN N N 6 _{YDU U &} ; : : : /       _H[S    O _{D D}     H[S_{ D O} N LN L ; U  7KXV

° ° ° ° ° ¯ ° ° ° ° ° ® ­ ¦  d ¦ d ¦   ! ¦   6 N & : ; ; LI ; U & U U D Y ; LI ; U & U U D Y 6 L N 6 N N N 1 L N LN L N LN L LN L L L N 1 L N LN L N LN L LN L L L           H[S    H[S        H[S    H[S       O D D D O D D D  7KH DERYH LV DQ RSWLPL]DWLRQ SUREOHP DQG WKH RSWLPDO

N ; DQG_{O FDQ EH VROYHG E\ QXPHULFDO PHWKRGV} $OJRULWKP  6WHS  6HW O  JXHVV ODQG VSHFLI\ [ WROHUDQFH 6WHS  &DOFXODWH O O   6ROYH : NE\ QXPHULFDO PHWKRGV     H[S    H[S     O 1 L YLDLUL DLN UL DLN &N UL DLN ; N O ¦    IRU DOO N N   « 6 6WHS  ,) _! N ; WKHQ VHW  N N ; ; (OVH VHW  N ; (QG ,) 6WHS  ,I _¦6 _N L & :   [ d ¦ d 6 N ; N _¦6 N L & :   [ WKHQ VWRS (QGLI 6WHS  ,I _¦6 N ; N _¦6 N L & :   [ WKHQ VHOHFW OO OO (OVH VHOHFW OO! OO (QGLI 6WHS  *R WR 6WHS  1RWH WKDW LI ULDLN&N ;N WKHQ ZH FDQ KDYH > @    |          H[S UL DLN &N ;N UL DLN &N ;N       N N LN L & ; U D  _

7KHUHIRUH IROORZ WKH VLPLODU DOJRULWKPLF SURFHGXUHV DV EHIRUH ZH FDQ REWDLQ

N

; DQG_{O }



&RQFOXVLRQV

,Q WKLV SDSHU ZH FRQVLGHU WKUHH NLQGV RI VRIWZDUH WHVWLQJUHVRXUFH DOORFDWLRQ SUREOHPV 7KH ILUVW SUREOHP LV WR PLQLPL]H WKH QXPEHU RI UHPDLQLQJ IDXOWV JLYHQ D IL[HG DPRXQW RI WHVWLQJHIIRUW DQG D UHOLDELOLW\ REMHFWLYH 7KH VHFRQG SUREOHP LV WR PLQLPL]H WKH DPRXQW RI WHVWLQJHIIRUW JLYHQ WKH QXPEHU RI UHPDLQLQJ IDXOWV DQG D UHOLDELOLW\ REMHFWLYH 7KH WKLUG SUREOHP PLQLPL]HV WKH FRVW JLYHQ WKH QXPEHU RI UHPDLQLQJ IDXOWV DQG D UHOLDELOLW\ REMHFWLYH :H SURSRVH VHYHUDO VWUDWHJLHV IRU PRGXOH WHVWLQJ LQ RUGHU WR KHOS VRIWZDUH SURMHFW PDQDJHUV WR VROYH WKHVH SUREOHPV DQG WR PDNH WKH EHVW GHFLVLRQV 1DPHO\ ZH SURYLGH VHYHUDO V\VWHPDWLF VROXWLRQV EDVHG RQ DQ 1+33 PRGHO DOORZLQJ WKHVH PDQDJHUV WR HDVLO\ DOORFDWH D VSHFLILHG DPRXQW RI WHVWLQJUHVRXUFH H[SHQGLWXUHV IRU HDFK VRIWZDUH PRGXOH XQGHU VRPH FRQVWUDLQWV :H GHVFULEH QXPHULFDO H[DPSOHV RQ WKH RSWLPDO WHVWLQJUHVRXUFH DOORFDWLRQ SUREOHPV WR VKRZ DSSOLFDWLRQV DQG LPSDFWV RI WKH SURSRVHG VWUDWHJLHV GXULQJ PRGXOH WHVWLQJ )LQDOO\ ZH H[WHQG RXU DSSURDFK WR LQWHJUDWLRQ WHVWLQJ IRU WKH WHVWLQJUHVRXUFH PDQDJHPHQW VWUDWHJLHV DQG SURYLGH FRPSOHWH DQDO\WLFDO VROXWLRQV IRU WKHP



$FNQRZOHGJHPHQW

7KLV UHVHDUFK ZDV VXSSRUWHG E\ WKH 1DWLRQDO 6FLHQFH &RXQFLO 7DLZDQ 52& XQGHU *UDQW 16& (  DQG DOVR VXEVWDQWLDOO\ VXSSRUWHG E\ D JUDQW IURP WKH 5HVHDUFK *UDQW &RXQFLO RI WKH +RQJ .RQJ 6SHFLDO $GPLQLVWUDWLYH 5HJLRQ 3URMHFW 1R &8+.( )XUWKHU ZH WKDQN WKH DQRQ\PRXV UHIHUHHV IRU WKHLU FULWLFDO UHYLHZ DQG FRPPHQWV



5HIHUHQFHV

>@ 0 5 /\X  +DQGERRN RI 6RIWZDUH 5HOLDELOLW\ (QJLQHHULQJ 0F*UDZ +LOO

>@ 2 %HUPDQ DQG 1 $VKUDIL Ø 2SWLPL]DWLRQ 0RGHOV IRU 5HOLDELOLW\ RI 0RGXODU 6RIWZDUH 6\VWHPVÙ,((( 7UDQV RQ 6RIWZDUH (QJLQHHULQJ YRO  QR  SS  1RY 

>@ ' : &RLW ³(FRQRPLF $OORFDWLRQ RI 7HVW 7LPHV IRU 6XEV\VWHP/HYHO 5HOLDELOLW\ *URZWK 7HVWLQJ´ ,,( 7UDQVDFWLRQV YRO  QR  SS ± 'HF >@ 3 .XEDW DQG + 6 .RFK ³0DQDJLQJ 7HVW3URFHGXUH WR

$FKLHYH 5HOLDEOH 6RIWZDUH´ ,((( 7UDQV RQ 5HOLDELOLW\ YRO  1R  SS  

>@ 3 .XEDW ³$VVHVVLQJ UHOLDELOLW\ RI 0RGXODU 6RIWZDUH´ 2SHUDWLRQ 5HVHDUFK /HWWHUV YRO  1R  SS   >@ + 2KWHUD DQG 6 <DPDGD ³2SWLPDO $OORFDWLRQ DQG &RQWURO

3UREOHPV IRU 6RIWZDUH7HVWLQJ 5HVRXUFHV´ ,((( 7UDQV RQ 5HOLDELOLW\ YRO  1R  SS  

>@ 6 <DPDGD 7 ,FKLPRUL DQG 0 1LVKLZDNL ³2SWLPDO $OORFDWLRQ 3ROLFLHV IRU 7HVWLQJ 5HVRXUFH %DVHG RQ D 6RIWZDUH 5HOLDELOLW\ *URZWK 0RGHO´ 0DWKHPDWLFDO DQG &RPSXWHU 0RGHOOLQJ 9RO  SS  

>@ < : /HXQJ ³'\QDPLF 5HVRXUFH $OORFDWLRQ IRU 6RIWZDUH 0RGXOH 7HVWLQJ´ 7KH -RXUQDO RI 6\VWHPV DQG 6RIWZDUH YRO 1R SS 0D\ 

>@ < : /HXQJ ³6RIWZDUH 5HOLDELOLW\ $OORFDWLRQ 8QGHU 8QFHUWDLQ 2SHUDWLRQDO 3URILOHV³ -RXUQDO RI WKH 2SHUDWLRQDO 5HVHDUFK 6RFLHW\ YRO 1R SS $SULO  >@ < : /HXQJ ³2SWLPDO 5HOLDELOLW\ $OORFDWLRQ IRU 0RGXODU

6RIWZDUH 6\VWHP 'HVLJQHG IRU 0XOWLSOH &XVWRPHUV´ ,(,&( 7UDQV RQ ,QIRUPDWLRQ DQG 6\VWHPV YRO 1R SS 'HFHPEHU 

>@ 5 + +XR 6 < .XR DQG < 3 &KDQJ ³(IILFLHQW $OORFDWLRQ RI 7HVWLQJ 5HVRXUFHV IRU 6RIWZDUH 0RGXOH 7HVWLQJ %DVHG RQ WKH +\SHU*HRPHWULF 'LVWULEXWLRQ 6RIWZDUH 5HOLDELOLW\ *URZWK 0RGHO´ ,((( 7UDQV RQ 5HOLDELOLW\ 9RO  1R SS  'HF 

>@ % <DQJ DQG 0 ;LH ³7HVWLQJ5HVRXUFH $OORFDWLRQ IRU 5HGXQGDQW 6RIWZDUH 6\VWHPV´ 3URFHHGLQJV RI WKH  3DFLILF 5LP ,QWHUQDWLRQDO 6\PSRVLXP RQ 'HSHQGDEOH &RPSXWLQJ 35'& 'HF  +RQJ .RQJ &KLQD >@ % <DQJ DQG 0 ;LH ³2SWLPDO 7HVWLQJWLPH $OORFDWLRQ IRU

0RGXODU 6\VWHPV´ ,QWHUQDWLRQDO -RXUQDO RI 4XDOLW\ DQG 5HOLDELOLW\ 0DQDJHPHQW 9RO  1R   >@ 0 5 /\X 6 5DQJDUDMDQ DQG $ 3 $ YDQ 0RRUVHO

³2SWLPDO $OORFDWLRQ RI 7HVW 5HVRXUFHV IRU 6RIWZDUH 5HOLDELOLW\ *URZWK 0RGHOLQJ LQ 6RIWZDUH 'HYHORSPHQW´ ,((( 7UDQV RQ 5HOLDELOLW\ 9RO  1R  SS  -XQH 

>@ 6 < .XR & < +XDQJ DQG 0 5 /\X ³)UDPHZRUN IRU 0RGHOLQJ 6RIWZDUH 5HOLDELOLW\ 8VLQJ 9DULRXV 7HVWLQJ(IIRUWV DQG )DXOW'HWHFWLRQ 5DWHV´ ,((( 7UDQV RQ 5HOLDELOLW\ 9RO  1R  SS  6HS 

>@ & < +XDQJ DQG 6 < .XR ³$QDO\VLV DQG $VVHVVPHQW RI ,QFRUSRUDWLQJ /RJLVWLF 7HVWLQJ (IIRUW )XQFWLRQ LQWR 6RIWZDUH 5HOLDELOLW\ 0RGHOLQJ´ WR DSSHDU LQ ,((( 7UDQV RQ 5HOLDELOLW\ 6HSW 

>@ & < +XDQJ - + /R 6 < .XR DQG 0 5 /\X ³6RIWZDUH 5HOLDELOLW\ 0RGHOLQJ DQG &RVW (VWLPDWLRQ ,QFRUSRUDWLQJ 7HVWLQJ(IIRUW DQG (IILFLHQF\´ ,((( 3URFHHGLQJV RI WKH WK ,QWHUQDWLRQDO 6\PSRVLXP RQ 6RIWZDUH 5HOLDELOLW\ (QJLQHHULQJ ,665( SS  1RY  %RFD 5DWRQ )/ 86$ >@ & < +XDQJ - + /R DQG 6 < .XR ³$ 3UDJPDWLF 6WXG\ RI

3DUDPHWULF 'HFRPSRVLWLRQ 0RGHOV IRU (VWLPDWLQJ 6RIWZDUH 5HOLDELOLW\ *URZWK´ 3URFHHGLQJV RI WKH WK ,QWHUQDWLRQDO 6\PSRVLXP RQ 6RIWZDUH 5HOLDELOLW\ (QJLQHHULQJ ,665( SS  1RY  3DGHUERUQ *HUPDQ\

>@ * / 1HPKDXVHU $ + * 5LQQRR\ .DQ 0- 7RGG ³2SWLPL]DWLRQ +DQGERRNV LQ 2SHUDWLRQV 5HVHDUFK DQG 0DQDJHPHQW 6FLHQFH Y´ 1RUWK+ROODQG

>@ ' 3 +H\PDQ DQG 0 - 6REHO ³6WRFKDVWLF 0RGHOV

+DQGERRNV LQ 2SHUDWLRQV 5HVHDUFK DQG 0DQDJHPHQW 6FLHQFH Y´ 1HZ <RUN 1< 1RUWK+ROODQG

>@ 0 6 %D]DUDD + ' 6KHUDOL DQG &0 6KHWW\  1RQOLQHDU 3URJUDPPLQJ 7KHRU\ DQG $OJRULWKPV QG HG -RKQ :LOH\ 6RQV

>@ - ' 0XVD $ ,DQQLQR DQG . 2NXPRWR  6RIWZDUH 5HOLDELOLW\ 0HDVXUHPHQW 3UHGLFWLRQ DQG $SSOLFDWLRQ 0F*UDZ +LOO

>@ 0 5 /\X 6 5DQJDUDMDQ DQG $ 3 $ YDQ 0RRUVHO ³2SWLPL]DWLRQ RI 5HOLDELOLW\ $OORFDWLRQ DQG 7HVWLQJ 6FKHGXOH IRU 6RIWZDUH 6\VWHPV´ 3URFHHGLQJV RI WKH WK ,QWHUQDWLRQDO 6\PSRVLXP RQ 6RIWZDUH 5HOLDELOLW\ (QJLQHHULQJ ,665( SS  1RY  $OEXTXHUTXH 1HZ 0H[LFR

>@ 5XVVHOO $ )LQN 5HOLDELOLW\ 0RGHOLQJ RI )UHHO\$YDLODEOH ,QWHUQHW'LVWULEXWHG 6RIWZDUH 3URFHHGLQJV RI WKH WK ,QWHUQDWLRQDO 6\PSRVLXP RQ 0HWULFV SS   >@ % /LWWOHZRRG ³6RIWZDUH 5HOLDELOLW\ 0RGHO IRU 0RGXODU

3URJUDP 6WUXFWXUH´ ,((( 7UDQV RQ 5HOLDELOLW\ YRO  1R  SS  

>@ 5 / %XOILQ DQG & < /LX ³2SWLPDO $OORFDWLRQ RI 5HGXQGDQW &RPSRQHQWV IRU /DUJH 6\VWHPV´ ,((( 7UDQV 5HOLDELOLW\ YRO 5 SS ± 

>@ ) =DKHGL DQG 1 $VKUDIL ³6RIWZDUH 5HOLDELOLW\ $OORFDWLRQ %DVHG RQ 6WUXFWXUH 8WLOLW\ 3ULFH DQG &RVWÙ,((( 7UDQV RQ 6RIWZDUH (QJLQHHULQJ YRO  QR  SS  $SULO