Sensitivity Analysis of Software Reliability for Component-Based Software Applications

(1)

Reliability assessment and sensitivity analysis of software

reliability growth modeling based on software module structure

Jung-Hua Lo

a

, Chin-Yu Huang

b

, Ing-Yi Chen

c

, Sy-Yen Kuo

d,*

, Michael R. Lyu

e

a_{Department of Information Management, Lan-Yang Institute of Technology, I-Lan, Taiwan} b_{Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan}

c_{Department of Computer Science and Information Engineering, National Taipei University of Technology, Taipei, Taiwan} d_{Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan}

e_{Department of Computer Science and Information Engineering, The Chinese University of Hong Kong, Hong Kong}

Received 15 December 2003; received in revised form 15 April 2004; accepted 15 June 2004 Available online 11 September 2004

Abstract

Software reliability is an important characteristic for most systems. A number of reliability models have been developed to eval-uate the reliability of a software system. The parameters in these software reliability models are usually directly obtained from the field failure data. Due to the dynamic properties of the system and the insufficiency of the failure data, the accurate values of the parameters are hard to determine. Therefore, the sensitivity analysis is often used in this stage to deal with this problem. Sensitivity analysis provides a way to analyzing the impact of the different parameters. In order to assess the reliability of a component-based software, we propose a new approach to analyzing the reliability of the system, based on the reliabilities of the individual compo-nents and the architecture of the system. Furthermore, we present the sensitivity analysis on the reliability of a component-based software in order to determine which of the components affects the reliability of the system most. Finally, three general examples are evaluated to validate and show the effectiveness of the proposed approach.

Keywords: Non-homogeneous Poisson process; Sensitivity analysis; Software reliability; Software reliability growth models

1. Introduction

Due to the recent rapid developments of computer and network technologies, the Internet and World Wide Web make it possible for users to access a variety of resources and applications distributed over the world. Since soft-ware is embedded in computer technologies and perme-ates our daily life, the correct performance of software systems becomes an important issue of many critical

sys-tems. Software designers are motivated to integrate com-mercial oﬀ-the-shelf (COTS) software components for rapid software development. To ensure high reliability for such applications using software components as their building blocks to construct a software system, dependa-ble components have to be deployed to meet the reliabil-ity requirements. Therefore, it is necessary to assess the reliabilities of such systems by investigating the architec-tures, the testing strategies, and the component reliabili-ties (Lyu, 1996; Musa et al., 1987; Musa, 1998). For instance,Everett (1999)uses the extended execution time (EET) reliability growth model and several test cases to model the reliability growth of each component.Gokhale et al. (1998)andGokhale and Trivedi (2002)assumes the failure behavior of each component is described by a

* _{Corresponding author. Tel.: +886 2 2368 9172/2363 525; fax:}

+889 2 2367 1909/886 2 2363 8247.

E-mail address:sykuo@cc.ee.ntu.edu.tw(S.-Y. Kuo).

(2)

time-dependent failure intensity. The total number of failures is obtained and the reliability is estimated via the enhanced non-homogeneous Poisson process (ENHPP). Krishnamurthy and Mathur (1999) evaluated a method, known as component based reliability estimation (CBRE), to estimate the reliability of a com-ponent-based software system using reliabilities of its components.Yacoub et al. (1999)proposed a reliability analysis technique called the scenario-based reliability analysis (SBRA), which is based on execution scenarios to derive a probabilistic model for the analysis of a com-ponent-based system. In the field of software reliability modeling,Musa (1994, 1998)andMusa et al. (1987)first investigated the validity of the execution time theory and operational profile. Recently, we (Huang et al., 2002;Lo et al., 2002, 2003) adopted the concept of testing-effort within an NHPP model to get a better description of the software fault phenomenon. The failure behavior of each component can be described by failure intensity. Thus, an appropriate software reliability growth model can be applied for estimating the component reliability from the failure data. However, due to the insufficiencyof failure data, sensitivity analysis of the reliability estima-tion has also been investigated in the context of black-box models (Musa, 1994; Chen et al., 1994; Pasquini et al., 1986). The parameters in these software reliability models are usually obtained from the failure data. Sensi-tivity analysis provides an approach to analyzing the im-pact of the parameters (Musa, 1993, 1994). In general, one difficulty in estimating the reliability of a system in the testing stage is the insufficiency of the failure data and therefore the accurate values of the parameters are hard to get. Sensitivity analysis is often used in this stage to deal with this problem. In this paper, we investigate the sensitivity analysis of the reliability for a component-based software system. The method is very useful in prac-tice. For example, if we use the approach to determine that a parameter or a component in a system is the most sensitive, it is critical for the software testing-team to have this parameter estimated as accurately as possible or allocate more resources for this component. The organization of this paper is as follows. Section 2 presents an analytical approach to estimating the reliability of a system. Three different architecture styles are conducted in Section 3. Sensitivity analysis is discussed in Section 4. Experimental results of sensitivity analysis of system reliability are depicted in Section 5. Conclusions are pre-sented in Section 6.

2. Reliability prediction for module software systems 2.1. Notations

Pr probability function

pow(x, y) power function, i.e. pow(x, y) = xy

Xn discrete-time Markov chain with some

absorb-ing and some transient states

Pij probability of Xn+1being in state j given Xnis in

state i, i.e. Pr(Xn+1= jjXn= i)

Nij number of visits to state j before entering an

absorbing state given X0= i

lij expected value of Nij, E(Nij)

gk probability of absorption when a process

termi-nates at an absorbing state k

hi expected value of the number of visits to

Com-ponent i

Ri reliability of Component i

Rs reliability of the system

Tp,hi relative change of the system reliability as hiis

changed by 100p%

Sp,hi sensitivity of the relative change of the system

reliability to hiwhen hiis changed by 100p%

Tp,Ri relative change of the system reliability as Riis

changed by 100p%

Sp,Ri sensitivity of the relative change of the system

reliability to Rias Riis changed by 100p%

pE_ij erroneous transition probability with respect to the estimated software usage used in the test pT_ij true transition probability regarding the true

software usage

eij error corresponding to pij, i.e. eij¼ pEij pTij

qij relative error, i.e. qij¼ eij=pTij

qi relative error of transition probability in

Com-ponent i, i.e. qi¼ qijpTij=ð1 pTijÞ

Tp,Pij relative change of the system reliability as p

T ijis

changed by 100p%

Sp,Pij sensitivity of the relative change of the system

reliability to pT

ij as pTij is changed by 100p%

Tp,qi relative change of the system reliability as qiis

changed by 100p%

Sp,qi sensitivity of the relative change of the system

reliability to qias qiis changed by 100p%

2.2. Reliability assessment

To construct a component-based software, it in-volves assembling components together, allowing plug-and-play in a collection of self-contained and loosely coupled components, and interactions among these integrated components (Lyu, 1996; Musa, 1998). Such an approach facilitates software construc-tion and has been increasingly adopted for software development. Due to the complex characteristics of a component-based software, the assessment of the reliability of a software system contains two major tasks: the estimation of the reliability of a compo-nent-based system and the estimation of the reliabili-ties of its individual components. The novelty of this integrated approach presented here lies in the fact that not only the failure behavior of each component

(3)

can be speciﬁed but also the architecture of a compo-nent-based software system can be modeled as a Markov process. For this approach, we have the fol-lowing assumptions:

• The failures of the software components are inde-pendent: A software system can be viewed as com-posed of logically independent components, which can be implemented and tested individually (Gokhale and Trivedi, 2002;Krishnamurthy and Mathur, 1997;

Lo et al., 2002). We assume in this paper that a soft-ware system contains a collection of softsoft-ware compo-nents, which may be developed or executed concurrently.

• The transfer of control among software components follows a Markov process: We assume that the exchanges of controls among these components are characterized according to the rules of a Markov process. The probability of transition form one com-ponent to another is determined from the operational proﬁle of a system. This indicates that the next trans-fer of control to be executed is independent of the past history and depends only on the present component. Under the assumption of independence among the successive executions of the components, the reliability of a system can be predicted by the path-based ap-proaches (Gokhale and Trivedi, 2002; Pasquini et al., 1986;Xie and Shen, 1998). That is, if the reliability of an individual component is assumed to be independent of others, the path reliability is the product of compo-nent reliabilities. Therefore, in this section, we propose an approach to estimating the reliability of a compo-nent-based system by taking the architecture of the soft-ware system and the reliabilities of the components into consideration. For example, if a system consists of n components with reliabilities denoted by R1, . . . , Rn

respectively, the reliability of an execution path, 1, 3, 2, 3, 2, 3, 4, 3, n, is given by R1· R2

2

· R3 4

· R4· Rn.

The architecture of a software system gives a description of how the individual components are expected to inter-act during system operations. In addition, we obtain an estimate of the reliability of the overall system as well as describe the corresponding mathematical properties explicitly in detail. In particular, we consider systems with diﬀerent architecture styles and utilize the Markov process to model the failure behaviors of the applica-tions. Three general input–output cases were employed. Finally, we develop three methodologies to estimating the reliability of a software system. The proofs of the following theorems can be found in our previous results (Lo et al., 2002).

Theorem 1. (Single-input/single-output system) Con-sider a single-input and single-output system consisting of N components with reliabilities R1, . . . , RN. Let {Xn} be

the Markov process where state N is an absorbing state, i.e. an output node, while states {1, 2, . . . , N 1} are transient states. In particular, assume state 1 is the input node. Therefore, we have the system reliability: Rs¼ R1 RNQN1_i¼2 fpowðRi;l_1iÞg,

Theorem 2. (Single-input/multiple-output system) Con-sider a single-input and r-output system consisting of N components with individual reliabilities denoted by R1, . . . , RN. Let {Xn} be the Markov process where

{N,N 1, . . . , Nr + 1} are absorbing states (i.e. r out-put nodes) and {1, 2, . . . , N r} are transient states. In particular, assume state 1 is the input node. Therefore, we have the reliability of the system:

Rs¼ R1 Y Nr i¼2 powðRi;l1iÞ YN k¼N rþ1 powðRj;gjÞ:

Theorem 3. (Multiple-input/multiple-output system) Con-sider an s-input and r-output system consisting of N components with reliabilities R1, . . . , RN. Let {Xn} be a Markov

proc-ess where {N, N1, . . . , Nr + 1} are absorbing states (i.e. r output nodes) and {1, 2, . . . , Nr} are transient states. In particular, assume states {1, 2, . . . , s} are the input nodes with probability p1, p2, . . . , ps respectively.

Therefore, the system reliability, Rsequals

Ys i¼1 powðRi; piÞ Y Nr j¼sþ1 pow Rj; Xs l¼1 p_ll_lj ! Y N k¼N rþ1 powðRk;gkÞ:

3. Illustrations for system reliability

The following examples adapted fromCheung (1980),

Gokhale and Trivedi (2002) are used to illustrate the three architecture cases discussed in Section 2. Without loss of generality, we use the terminating application reported in Gokhale and Trivedi (2002) as a running example and let the estimated reliabilities of the compo-nents be regarded as unchanged throughout the follow-ing three subsections and listed in Table 1.

3.1. Example 1: a single-input/single-output type

The ﬁrst example is a single-input/single-output sys-tem. It consists of 10 components where Component 1 is the input component and Component 10 the output component. Fig. 1 depicts the control-ﬂow graph of the example, and the transition probabilities among the components are given as follows: P1,2= 0.6,

P1,3= 0.2, P1,4= 0.2, P2,3= 0.7, P2,5= 0.3, P3,5= 1.0,

(4)

P6,7= 0.3, P6,8= 0.1, P6,9= 0.3, P7,2= 0.5, P7,9= 0.5,

P8,4= 0.25, P8,10= 0.75, P9,8= 0.1, P9,10= 0.9.

There-fore, according to Theorem 1, the expected number of visits on each transient state before absorption from the input node (Component 1) and the probability of absorption can be derived as listed in Table 1. Thus, the system reliability is estimated as R1= 0.8482.

3.2. Example 2: a single-input/multiple-output type In this example, we delete two links of the original program control graph in Example 1, and obtain the modiﬁed graph as shown inFig. 2. The modiﬁcation is a simple transformation from a single-output system to a multiple-output system and the corresponding tran-sition probabilities are similar to Example 1. Again, fol-lowing the same approach in Theorem 2, we can have following results as listed in Table 1. To go a further step, the reliability of the application, R2is 0.8890.

3.3. Example 3: a multiple-input/multiple-output type In this example, the process will start from one of the two input components (Components 1 and 2) with equal probability and terminates at the output components (Components 7 and 10). That is, the modiﬁcation is a transformation from a single-input system to a multi-ple-input system. Fig. 3 depicts the control-ﬂow graph of the example and the transition probabilities are

simi-lar to Example 2 except P1,3= 0.5 and P1,4= 0.5. As a

result, according to Theorem 3, portion of the vector of weights can be obtained by hk ¼P

2

l¼1plllk.

There-fore, we have (h1, h2, h3, h4, h5, h6, h8, h9) = (0.5, 0.5,

0.673, 0.4057, 0.9853, 0.2434, 0.6228, 0.073). On the other hand, with the aim to computing the probability of absorption at each absorbing state, the following infor-mation about the two absorbing states is obtained based on Theorem 3: h7= 0.4672, h10= 0.5327. Thus, we have

the reliability of the system is R3= 0.8929.

1 2 4 3 5 7 6 8 9 10

Fig. 1. A single-input/single-output system. Table 1

The estimated component reliabilities (Ri) and the expected values of the number of visits

1 2 3 4 5 6 7 8 9 10 Ri 0.99 0.98 0.99 0.96 0.98 0.95 0.98 0.96 0.97 0.99 hiin Example 1 1 1.4717 1.3254 0.5289 1.9784 0.3173 1.7433 1.3155 0.9669 1 hiin Example 2 1 0.6 0.6845 0.3581 1.0077 0.2149 0.6326 0.0645 0.4676 0.5324 hiin Example 3 0.5 0.5 0.673 0.4057 0.9853 0.2434 0.4672 0.6228 0.073 0.5327 1 2 4 3 5 7 6 8 9 10

Fig. 2. A single-input/multiple-output system.

0.5 0.5 1 2 4 3 5 7 6 8 9 10

(5)

4. Sensitivity analysis

The reliability of a component-based software system is often higher through the improvement of some compo-nents in the system. Therefore, considering such a system we are often interested to know which component is more important than others. Thus, the improvement of that important component will increase the system reliability more than others. Sensitivity analysis gives an approach to analyze the relative importance of input model param-eters in determining the value of an assigned output value (Xie and Shen, 1989). That is this method can help make a reasonable decision for this problem. Furthermore, a number of software reliability models have been devel-oped to evaluate the reliability of a system. The parame-ters in these models are usually obtained from the field failure data. In general, one difficulty in estimating the reliability of a system in the testing stage is the insuffi-ciency of failure data that makes the exact values of the parameters hard to get. Sensitivity analysis is often used in this stage due to the deviations of parameters (Musa, 1993, 1994;Xie and Shen, 1998). That is, sensitivity anal-ysis can help in investigating the effect of the uncertainty in parameters on the reliability estimated from model. In this paper, we will study the sensitivity analysis of the reli-ability of the component-based software applications in order to know which of the components affects the relia-bility of the system most. Consequently, from Theorems 1–3 in Section 2, the reliability of this system can be expressed as the general form:

Rs¼

YN i¼1

powðRi;hiÞ; ð1Þ

where Ri is the estimated reliability of Component i

and hi is the expected value of the number of visits to

Component i. Here Rscan be regarded as a function of

parameters Riand hi, i = 1, . . . , N. From the above

dis-cussion, in order to estimate the parameters in Eq. (1)

by using the hierarchical approach, it is necessary for software testers to know the information regarding a par-ticular application: architecture of the application (struc-ture of component interactions), software usage proﬁle (the exchange of controls among components deter-mined by transition probabilities), and component fail-ure behaviors (component reliabilities or failure intensity). However, the estimates may not always be accurate, especially in the early stage of the testing phase when a limited amount of information is available. Therefore, it is essential to know the sensitivity of re-quired knowledge regarding the estimated parameters.

4.1. The most sensitive parameter

Considering the parameter hi of Eq.(1), it would be

helpful to know which of the parameters aﬀects the

reli-ability of the system most, so that more accurate meas-urements can be made for the most important one (Gokhale and Trivedi, 2002). That is, we are concerned whether the condition of the following formula is sufficed: oRs ohi P oRohjs ; for all j ¼ 1; 2; . . . ; N: ð2Þ In practice, the frequency of a component being exe-cuted affects the overall system reliability. A higher fre-quency indicates a greater effect of that component on the performance of the system. This fact shows that the components should have distinct weights accord-ing to the architecture of the software system. In other words, the change in the intercomponent transi-tion probabilities of the software architecture mani-fests the change in the parameter hi of Eq. (1). On

the other hand, deﬁne Tp,hi as the relative change of

the system reliability, Rs, when hi is changed by

100p%. That is Tp;hi ¼ jRsðh1; . . . ;hiþ phi; . . . ;hNÞ Rsðh1; . . . ;hNÞj Rsðh1; . . . ;hNÞ 100%: ð3Þ Comparatively, let Sp,hi be the sensitivity of the

rela-tive change of the system reliability to hiwhen hiis

chan-ged by 100p% as the ratio of relative change for the two quantities. That is Sp;hi ¼ DRs=Rs Dhi=hi ¼Tp;hi p 100%: ð4Þ

Then, from Eqs. (2)–(4), the sensitivity analysis can be conducted and the most sensitive parameter, hi, in

discrete situation can also be found. That is

Sp;hi P Sp;hj; for all j¼ 1; 2; . . . ; N : ð5Þ

Therefore, we have the desired results: if power (Rj, hj) P power(Ri, hi), for all j = 1,2, . . . , N., then hi is

the most sensitive parameter.

4.2. The most sensitive component reliability

Similar to the reasoning in Section 4.1, for comparing the estimated component reliability Ri in Eq. (1), it

would be useful to know which of the components af-fects the reliability of the system most. That is, we are concerned whether the condition of the following for-mula is suﬃced: oRs oRi 6 oRs oRj ; for all j ¼ 1; 2; . . . ; N: ð6Þ Again, deﬁne Tp,Rias the relative change of the

sys-tem reliability, Rs, when Ri is changed by 100p%, and

(6)

the system reliability to Riwhen Riis changed by 100p%, that is, Tp;Ri ¼ jRsðR1; . . . ; Riþ pRi; . . . ; RNÞ RsðR1; . . . ; RNÞj RsðR1; . . . ; RNÞ 100%; ð7Þ Sp;Ri¼ Tp;Ri=p 100%: ð8Þ

Thus, the sensitivity analysis with respect to the relative change of component reliability can be performed and the most sensitive component, Ri, in discrete situation

can also be found. That is

Sp;RiP Sp;Rj; for all j¼ 1; 2; . . . ; N : ð9Þ

According to Eq.(9), we have the result that the one with the maximum parameter value is the most sensitive.

4.3. The most sensitive transition ﬂow

In Section 4.1, we have found the way to deal with the sensitive parameter problem in Eq. (1). Here, we will work on the sensitivity analysis of system reliability resulting from the relative change of transition probabil-ity. For a component-based software, different users will have different reliability performances, because they use the system in various ways or use different parts of the system. This dynamic knowledge about the probabilities for different uses in a component-based software is determined by the transition probabilities and appar-ently depends on the software usage, i.e. operational profile. In general, the operational profile is an estimated description of how the system will be used. One can characterize the usage by the operational profile, the set of operations available on the system and their asso-ciated probabilities of occurrences (Musa, 1993, 1994). In order to study the sensitivity of the system reliability to an error in one of the transition probability in the software usage, the method is carried out and the fol-lowing definitions and symbols are used as follows. Sup-pose the transition probability, pij, is incorrect, and let eij

be the error. Therefore, we have eij¼ pEij p

T

ij; ð10Þ

where pE

ijis the erroneous transition probability with

re-spect to the estimated software usage used in the test, and pT

ijindicates the true transition probability regarding

the true software usage. And let qijbe the relative error,

qij¼ eij=pTij. To go a step further, from the property of

Markov process, we have PN_j¼1pT

ij¼ 1 and

PN

j¼1pEij¼ 1

for Component i. In particular, we assume that qijdoes

not have an eﬀect on other qikso that they all have the

same relative error qi. Therefore, from Eq. (10),

PN j¼1pTij¼ 1, and PN j¼1pEij¼ 1 we have XN j¼1 eij ¼ qijp T ijþ XN k¼1 k6¼j qikp T ik ¼ qijp T ijþ qi XN k¼1 k6¼j pT_ij ¼ 0: ð11Þ Finally, we obtain qi¼ qijpTij=ð1 p T ijÞ

4.3.1. The most sensitive interaction

Afterward, we can deﬁne Tp,Pijas the relative change

of the system reliability when the transition probability pT

ij is changed by 100p%, and also let Sp,Pijbe the

sensi-tivity of the relative change of the system reliability to pT ij

when pT

ij is changed by 100p%, that is,

Tp;Pij¼ jRsðPT11; . . . ; P T ijþ pP T ij; . . . ; P T NNÞ RsðPT11; . . . ; P T NNÞj RsðPTij; . . . ; P T ijÞ 100%; ð12Þ Sp;Pij ¼ Tp;Pij=p 100%: ð13Þ

Thus, the sensitivity analysis with respect to the relative change of transition probability can be conducted and the most sensitive interaction between components can be found.

4.3.2. The most sensitive relative error component On the other hand, we deﬁne Tp,qi as the relative

change of the system reliability when the relative error of transition probability in Component i, qi, is changed

by 100p%, and also let Sp,qibe the sensitivity of the

rel-ative change of the system reliability to qi when qi is

changed by 100p%, that is,

Tp;qi ¼ jRsðq1; . . . ;qiþ pqi; . . . ;qNÞ Rsðq1; . . . ;qNÞj Rsðq1; . . . ;qNÞ 100%; ð14Þ Sp;qi¼ Tp;qi=p 100%: ð15Þ

Thus, the sensitivity analysis with respect to the relative change of the relative error of transition probability in one component can be conducted and the most sensitive relative error in component can be found.

5. Experimental results for sensitivity analysis 5.1. The most sensitive parameter

As for the example of the software architecture with single-input/single-output in Section 3.1, we apply the results listed in Table 1(the component reliability and the estimated expected visits for each component, i.e. l1i) to Eq.(5). After some computations, we can ﬁgure

(7)

more than the other so that more accurate estimates can be obtained for the most important one. In this case, the parameter of Component 8 (l18) is the most sensitive

parameter because it has the minimum value (pow(0.96, 1.3155) = 0.9477) than others. Furthermore, the relationship between the sensitivity, Sp,hi, and the

rel-ative change of component parameter hi is depicted in

Fig. 4. In order to present the importance of each parameter, the curves in the ﬁgures are ordered by its

sensitivity decreasingly. We also apply the same ap-proach to Example 2 and Example 3 and conclude that the parameter of Component 8 is the most sensitive parameter as shown inFigs. 5 and 6.

5.2. The most sensitive component reliability

Similarly, we use the estimated vector, l1ifor Eq.(9).

Because Component 5 has the maximum parameter

Fig. 5. The most sensitive parameter in Example 2.

Fig. 4. The most sensitive parameter in Example 1. Fig. 6. The most sensitive parameter in Example 3.

(8)

value (l15= 1.9784), thus from the result in Section 4.2

we know Component 5 is the most sensitive. As for Example 2, the parameter value of Component 5 is 1.0077 and is larger than the others. That is Component 5 is the most sensitive in Example 2. For Example 3, the most sensitive component is also Component 5 because its parameter value is 0.9853 and is the largest.Figs. 7–9

illustrates the relationships between the sensitivity, Sp,Ri,

and the relative change of component reliability in Examples 1–3.

5.3. The most sensitive interaction

Fig. 10depict the relationships between the sensitiv-ity, Sp,Pij, and the relative change of Pij for Example 1.

This figure presents the first six sensitive interactions and the first one is the most sensitive. For example, the transition from Component 8 to Component 4 is the most sensitive in these three examples. In particular, a 10% change of P84will imply a 0.39% change of the

system reliability in Example 1. This means that it is

Fig. 8. The most sensitive component reliability in Example 2.

Fig. 9. The most sensitive component reliability in Example 3.

Fig. 10. The most sensitive transition in Example 1.

(9)

much more important to obtain an accurate estimate of P84 than others. Moreover, Figs. 11–13 also give a

description about the relative change of system reliabil-ity as the change of the ﬁrst six sensitive transition prob-ability in Examples 1–3.

5.4. The most sensitive relative error component

Fig. 14 shows the result about the relationships be-tween the sensitivity, Sp,qi, and the relative change of

the relative error, qiin Example 1. Besides, the most

sen-sitive relative error of transition probability in Examples 1–3 is Component 8. For example, a 10% change of q8

will imply a 0.63% change of the system reliability in Example 2 as shown in Table 4 and Fig. 15. Finally, we also list the results for all of the most sensitivity case in Tables 2–4.

6. Conclusions

In this paper we have presented an approach for assessing the reliability of a component-based software.

Fig. 13. Reliability vs a change of transition probability in Example 3.

Fig. 14. The most sensitive relative error in Example 1.

Fig. 15. The most sensitive relative error in Example 2. Fig. 12. Reliability vs a change of transition probability in Example 2.

(10)

This methodology assumes that the software compo-nents are heterogeneous and the transfers of control be-tween components follow a discrete time Markov process. Besides, we also present the sensitivity analysis on the reliability of a component-based software in or-der to determine which of the components affects the reliability of the system most. Sensitivity analysis pro-vides a way to analyzing the impact of the parameters. In particular, we define several metrics on how to assess the most sensitive parameter in a system and derive some useful mathematical properties for the sensitivity analysis of system reliability. Finally, three different

architecture styles are utilized to validate the proposed approach. For the future works, we will focus on topics including comparisons with diﬀerent approaches, sensi-tivity analysis of resource allocation problems, and other sensitivity of software attributes.

Acknowledgments

This research was supported by the National Science Council, Taiwan, ROC, under Grant NSC 91-2213-E-002-042 and 93-2213-E-267-001, and also substantially

Table 2

The most sensitive cases in Example 1 The most sensitive

parameter case

The most sensitive component reliability case

The most sensitive interaction case

The most sensitive relative error case p% Rs Tp/2,h8 Sp/2,h8 Rs Tp/2,R5 Sp/2, R5 Rs Tp,P8,4 Sp,P8,4 Rs Tp,q8 Sp,q8 20 0.8452 0.0036 0.0356 0.9659 0.2010 1.3693 0.8413 0.0081 0.1160 0.8271 0.0249 0.1244 15 0.8460 0.0027 0.0356 0.9423 0.1371 1.3647 0.8430 0.0061 0.1148 0.8335 0.0174 0.1206 10 0.8467 0.0018 0.0356 0.9188 0.0920 1.3600 0.8447 0.0039 0.1136 0.8395 0.0103 0.1171 5 0.8475 0.009 0.0357 0.8955 0.0213 1.3554 0.8463 0.0022 0.1117 0.8451 0.0036 0.1131 5 0.8488 0.0009 0.0357 0.8265 0.0123 1.3459 0.8495 0.0015 0.0999 0.8531 0.0024 0.0980 10 0.8490 0.0018 0.0357 0.8039 0.0609 1.3411 0.8511 0.0035 0.1123 0.8580 0.0053 0.1026 15 0.8503 0.0027 0.0357 0.7814 0.1302 1.3363 0.8526 0.0056 0.1110 0.8628 0.0084 0.1040 20 0.8513 0.0036 0.0357 0.7590 0.1762 1.3314 0.8541 0.0078 0.1099 0.8650 0.0144 0.1060 Table 3

The most sensitive cases in Example 2 The most

sensitive parameter case

The most sensitive relative error case p% Rs Tp/2,h8 Sp/2,h8 Rs Tp/2,R5 Sp/2,R5 Rs Tp,P8,4 Sp,P8,4 Rs Tp,q8 Sp,q8 20 0.8867 0.0026 0.0258 0.9598 0.0796 1.0080 0.8994 0.0124 0.0593 0.8774 0.0131 0.0623 15 0.8873 0.0019 0.0258 0.9401 0.0575 1.0079 0.8970 0.0093 0.0584 0.8804 0.0096 0.0606 10 0.8879 0.0013 0.0258 0.9204 0.0353 1.0078 0.8945 0.0062 0.0570 0.8834 0.0063 0.0585 5 0.8884 0.0006 0.0258 0.9006 0.0131 1.0078 0.8920 0.0031 0.0537 0.8862 0.0031 0.0544 5 0.8896 0.0006 0.0258 0.8612 0.0312 1.0077 0.8866 0.0026 0.0572 0.8913 0.0026 0.0567 10 0.8901 0.0013 0.0258 0.8415 0.0534 1.0076 0.8838 0.0060 0.0627 0.8938 0.0054 0.0565 15 0.8907 0.0019 0.0258 0.8218 0.0756 1.0075 0.8808 0.0090 0.0611 0.8967 0.0087 0.0593 20 0.8913 0.0026 0.0259 0.8021 0.0977 1.0074 0.8778 0.0119 0.0601 0.8992 0.0114 0.0578 Table 4

The most sensitive cases in Example 3 The most

sensitive parameter case

The most sensitive relative error case p% Rs Tp/2,h8 Sp/2,h8 Rs Tp/2,R5 Sp/2,R5 Rs Tp,P8,4 Sp,P8,4 Rs Tp,q8 Sp,q8 20 0.8906 0.0025 0.0254 0.9808 0.0985 0.9846 0.8891 0.0120 0.0572 0.8816 0.0126 0.1244 15 0.8912 0.0019 0.0254 0.9588 0.0739 0.9848 0.8901 0.0090 0.0560 0.8846 0.0093 0.1206 10 0.8918 0.0013 0.0254 0.9369 0.0492 0.9850 0.8910 0.0059 0.0540 0.8874 0.0061 0.1171 5 0.8923 0.0006 0.0254 0.9149 0.0246 0.9851 0.8918 0.0030 0.0493 0.8902 0.0030 0.1131 5 0.8935 0.0006 0.0254 0.8709 0.0246 0.9855 0.8936 0.0024 0.0611 0.8950 0.0023 0.103 10 0.8940 0.0013 0.0254 0.8489 0.0493 0.9857 0.8944 0.0058 0.0646 0.8976 0.0053 0.104 15 0.8946 0.0019 0.0254 0.8269 0.0739 0.9859 0.8953 0.0087 0.0620 0.9004 0.0084 0.106 20 0.8952 0.0025 0.0255 0.8049 0.0986 0.9861 0.8961 0.0115 0.0606 0.9028 0.0111 0.105

(11)

supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, Chi-na (Project no. CUHK4360/02E).

References

Chen, M.H., Mathur, A.P., Rego, V.J., 1994. A case study to investigate sensitivity of reliability estimates to errors in opera-tional proﬁle. In: Proceedings of the 5th Internaopera-tional Symposium on Software Reliability Engineering, October, pp. 276–281. Cheung, R.C., 1980. A user-oriented software reliability model. IEEE

Trans. Softw. Eng. 6, 118–125.

Everett, W., 1999, Software component reliability analysis. In: Proceedings of the International Symposium on Application-Speciﬁc Systems and Software Engineering &Technology, March, pp. 204–211.

Gokhale, S.S., Trivedi, K.S., 2002. Reliability prediction and sensitiv-ity analysis based on software architecture. In: Proceedings of the 13th International Symposium on Software Reliability Engineer-ing, November, pp. 64–75.

Gokhale, S.S., Wong, W.E., Trivedi, K.S., Horgan, J.R., 1998. An analytic approach to architecture-based software reliability predic-tion. In: Proceedings of the International Symposium on Perform-ance and Dependability, September, pp. 13–22.

Krishnamurthy, S., Mathur, A.P., 1997. On the estimation of reliability of a software system using reliabilities of its component. In: Proceedings of the 8th International Symposium on Software Reliability Engineering, November, pp. 146–155.

Lo, J.H., Kuo, S.Y., Lyu, M.R., Huang, C.Y., 2002. Optimal resource allocation and reliability analysis for component-based software applications. In: The 26th Annual International Computer Soft-ware and Applications Conference, August, pp. 7–12.

Lo, J.H., Huang, C.Y., Kuo, S.Y., Lyu, M.R., 2003. Sensitivity analysis of software reliability for component-based software applications. In: The 27th Annual International Computer Soft-ware and Applications Conference, November, pp. 500–505. Lyu, M.R., 1996. Handbook of Software Reliability Engineering.

McGraw-Hill.

Musa, J.D., 1993. Operational proﬁles in software reliability engineer-ing. IEEE Trans. Softw. 10, 14–32.

Musa, J.D., 1994. Sensitivity of ﬁeld failure intensity to operational proﬁle errors. In: Proceedings of the 5th International Symposium on Software Reliability Engineering, October, pp. 334–337. Musa, J.D., 1998. Software Reliability Engineering: More Reliable

Software, Faster Development and Testing. McGraw-Hill. Musa, J.D., Iannino, A., Okumoto, K., 1987. Software Reliability,

Measurement, Prediction and Application. McGraw-Hill. Pasquini, A., Crespo, A.N., Matrella, P., 1986. Sensitivity of

reliabil-ity-growth model to operational proﬁle errors vs. testing accuracy. IEEE Trans. Reliab. 45, 531–540.

Xie, M., Shen, K., 1989. On ranking of system components with respect to diﬀerent improvement actions. Microelectron. Reliab. 29, 159–164.

Xie, M., Shen, G.Y., 1998. A study of the sensitivity of software release time. J. Syst. Softw. 44, 163–168.

Yacoub, S.M.B., Cukic, Ammar, H.H., 1999. A component-based approach to reliability analysis of distributed systems. In: Pro-ceedings of the 18th International Symposium on Reliable Distributed Systems, pp. 158–167.

Jung-Hua Lo is currently an Assistant Professor in the Department of Information Management, LanYang Institute of Technology, I-Lan, Taiwan. Dr. Lo received the BS (1993) in Mathematics from National Taiwan University and the MS (1995) and the PhD (2003) in Electrical Engineering from National Taiwan University. His research interests are software reliability and testing.

Chin-Yu Huang is currently an Assistant Professor in the Department of Computer Science, National Tsing Hua University, Hsinchu, Tai-wan. Dr. Huang received the MS (1994) and the PhD (2000) in Elec-trical Engineering from National Taiwan University, Taipei. He was with the Bank of Taiwan from 1994 to 1999 and was a senior software engineer at Taiwan Semiconductor Manufacturing Company from 1999 to 2000. Prior to joining Tsing Hua University, he was a division chief in the department of planning and design, information systems oﬃce, the Central Bank of China, Taipei, from 2000 to 2003. His research interests are software engineering, software reliability and testing, and software fault-tolerance, etc.

Ing-Yi Chen received a BS in physic from National Central University, Taiwan, in 1984, and MS and PhD degrees in Electrical and Computer Engineering from the University of Arizona, USA, in 1989 and 1992, respectively. Dr. Chen is currently an associate professor in Computer Science and Information Engineering Department, National Taipei University of Technology, Taipei, Taiwan. Prior to joining NTUT, he served as chief technology oﬃcer for Chinatimes from 2000 to 2001 with responsibility for the corporationÕs strategic technology planning and external technical aﬃliations with universities. Previously, he was a faculty member in the Department of Electronic Engineering at Chung Yuan Christian University, Taiwan between 1992 and 2000. His research interests include various topics in integration of web appli-cations and services, corporate evolving web uses, internet application layer protocols, database systems, web site security, and web site per-formance management. A member of the Phi Tau Phi scholastic honor society, Dr. Chen was a recipient of the IEEE/ACM Design Auto-mation Scholarship Award in 1991, and the Distinguished Teaching Award at CYCU in 1996.

Sy-Yen Kuo received the BS (1979) in Electrical Engineering from National Taiwan University, the MS (1982) in Electrical and Com-puter Engineering from the University of California at Santa Barbara, and the PhD (1987) in Computer Science from the University of Illi-nois at Urbana-Champaign. Since 1991 he has been with National Taiwan University, where he is currently a professor and the Chairman of Department of Electrical Engineering. He spent his sabbatical year as a visiting researcher at AT&T Labs-Research, New Jersey from 1999 to 2000. He was the Chairman of the Department of Computer Science and Information Engineering, National Dong Hwa University, Taiwan from 1995 to 1998, a faculty member in the Department of Electrical and Computer Engineering at the University of Arizona from 1988 to 1991, and an engineer at Fairchild Semiconductor and Silvar-Lisco, both in California, from 1982 to 1984. In 1989, he also worked as a summer faculty fellow at Jet Propulsion Laboratory of California Institute of Technology. His current research interests include mobile computing and networks, dependable distributed sys-tems, software reliability, and optical WDM networks.

Professor Kuo is an IEEE Fellow. He has published more than 200 papers in journals and conferences. He received the distinguished research award (1997–2005) from the National Science Council, Tai-wan. He was also a recipient of the Best Paper Award in the 1996 International Symposium on Software Reliability Engineering, the Best Paper Award in the simulation and test category at the 1986 IEEE/ACM Design Automation Conference (DAC), the National Science FoundationÕs Research Initiation Award in 1989, and the IEEE/ACM Design Automation Scholarship in 1990 and 1991. Michael Rung-Tsong Lyu received the BS (1981) in Electrical Engi-neering from National Taiwan University, the MS (1985) in Computer Engineering from University of California, Santa Barbara, and the PhD (1988) in Computer Science from University of California, Los Angeles. He is a Professor in the Computer Science and Engineering Department of the Chinese University of Hong Kong. He worked at the Jet Propulsion Laboratory, Bellcore, and Bell Labs, and taught at the University of Iowa. His research interests include software relia-bility engineering, software fault tolerance, distributed systems, image and video processing, web technologies, multimedia systems, and wireless communications. He has published over 150 papers in these areas. He initiated International Symposium on Software Reliability Engineering (ISSRE), and was Program Chair for ISSREÕ1996, Pro-gram Co-Chair for WWW10, and General Chair for ISSREÕ2001. He also received Best Paper Awards in ISSREÕ98 and in ISSREÕ2002. He is the editor for two book volumes: Software Fault Tolerance, published by Wiley in 1995, and the Handbook of Software Reliability Engi-neering, published by IEEE and McGraw-Hill in 1996. He has been an associated editor of IEEE Transactions on Reliability, IEEE Trans-actions on Knowledge and Data Engineering, and Journal of Infor-mation Science and Engineering. Dr. Lyu is a fellow of IEEE.