Requirement Prioritization And Regression Testing

Published Date: 02 Nov 2017

Introduction

Requirement prioritization creates implementation order of elicited requirements, to deliver software in increments, satisfying its diverse stakeholders. During implementation of next increment, previously delayed highest priority requirements along with new or modified ones are selected in accordance of their priorities leading to modification of source code. Whenever source code is modified, newly added requirements are to be tested along with those sections of code that are affected as a result of changes in other parts (ripple effects). The effective prioritization process will yield a correct implementation order, agreed by diverse stakeholders, thus enhancing the project acceptability among them. This makes tester confident that software that is being under test, is the true image of stakeholder’s wishes. Next, the tester has to perform regression testing on these prioritized requirements with minimal number of test cases having high fault detection capability. Requirement priorities can filter out those test cases that are not associated with the requirements of the highest priorities. Further prioritization and selections can be employed on filtered reduced test suite. Testing of modified code against test cases associated with requirements of highest priorities ensures that "Most favored" functionality is tested earlier and thoroughly under resource constraints. Thus under resource constraints, regression testing effort can be minimized (keeping fault detection rate high) if modified code implements requirements that are prioritized using effective prioritization process. The objective of this chapter is to investigate/analyze the impact of requirement process and hence requirement priorities on regression testing.

Requirement Prioritization and Regression Testing

Requirements are prioritized by evaluating each requirement against multiple and possibly conflicting decision aspects or prioritization factors by diverse stakeholders with multiple views or roles, geographically distributed across organizational or country boundaries. The final priority assigned to particular requirement must be agreed by all stakeholders involved, so as to create win-win situation among them. Regression testing involves re-execution of modified code against already available test cases and newly added ones. Newly added test cases executes added requirements i.e. source code. Existing ones executes already tested code to ensure that the modification does not result in ripple effects. Execution of all available test cases is not feasible due to resource constraints. Thus minimizing the size of the test suite without hampering fault detection capability is one of main activity involved during regression testing. There are various approaches for reducing the size of the test suite like test suite selections, prioritization and minimizations. Once test suite is reduced to a manageable number, they are executed on modified code. For mismatch between actual output and received output, faults are traced in the code, corrected and failed test cases are re-executed.

Effect of Requirement Prioritization on Hybrid regression Testing

Let R be the set of "N" requirements ordered in increasing order of priorities i.e. R = {R1, R2, ......., RN}. Let T be the set of test cases i.e. T = {T1, T2, ...... , Tm}, where m>=n. In the first release, let us consider that the set of the highest priority requirements be given by R’ = {Rx, Ry, ......, Rp} are implemented. This set is tested by T’, set of associated test cases i.e. T’ = {Tx, Ty, ......., Tq}, |R’|=p, and q>=p. During next increment, software development team might have to implement some postponed highest priority requirements along with those implemented in previous versions that are modified this time. Let the requirements to be implemented in this release be given by R’’, R’’ = {Ri, Rj, ..... , Rk}. R’’ = {R|R is either changed requirement or requirement of highest priority, that was not implemented in previous release}. Let set R’’ be tested with test cases represented by T’’. Implementation of these requirements will result in changes and additions in the source code, requiring the need for regression testing. Software testing team will try to ensure the re-execution of modified code with minimal number of test cases, by employing some well known regression testing techniques. Test suite i.e. TR contains the entire test cases that are available to the testing team for testing this modified source code, and given by TR = {T’, T’’}, T’’ are newly added test cases while T’ are those test cases that executed previous release of this software. The effective regression testing technique will reduce the size of TR to a manageable number, such that reduction in the size does not affect the high fault detection capability of the test suite. Thus after employing the proper regression testing technique, TR gets reduced to TR = {T1’, T’’}, where T1’ is a subset of T’. Priorities assigned to the requirement during requirement engineering phase can aid in selection of subset T1’. The percentage savings in test cases are given by Eq 5.1:

Tsavings = [(T’- T1’)/ T1’]*100 (5.1) (1)

T1’ is the subset obtained by employing requirement priorities as "filter" during regression testing. Cardinality of this subset is directly proportional to assigned priorities i.e. Rp since it will house only those test cases that are associated with requirements with priorities above threshold value. The cardinality of the set i.e. T’ is a function of the total number of requirements i.e. N. Thus Eq 5.1 can be visualized as (Cx are constants):

Tsavings = [(F(N)- F(Rp))/ F(Rp)]*100 (5.2) (2)

Equation 5.2 shows that factor Tsavings is the function of Rp, and total number of requirements to be implemented i.e. N. Let us denote the fault detection capability of the set (T’) be FDR’ while FDR’’ for the subset (T1’). There might be change in fault detection capability, given by equation 5.3:

FDRchange = FDR’’ - FDR’ (5.3) For ideal or good regression testing, FDRchange should get equated to 0, thus equation 5.3 gets reduced to:

FDR’ = FDR’’ (5.4) Since the fault detection capability or the fault detection rate is dependent on test case selected for execution, thus FDR’ can be written as:

FDR’’ = F(T1’ + T’’) (5.5) (5)

FDR’’ = F(F(Rp) + T’’) (5.6) (6)

From equation 5.6 it is clear that Fault detection rate of prioritized set is directly related to requirement priorities. This equation 5.2 and 5.6 show that requirement priorities can influence both percentage savings in test cases (regression testing effort) and fault detection rate.

Experiments and Results

Objective

The objective of this chapter is to investigate the effects/ impact of the requirement prioritization process and hence of requirement priorities on regression testing of requirement priority on regression testing. By focusing on test cases associated with requirements of highest priorities, and further prioritizing them against some prioritization criteria’s, one can thoroughly test "most favored" software requirements against a minimal number of test cases. This thesis tries to answer following research questions:

How much the requirement prioritization process impacts regression testing?

How much the requirement priorities impacts regression testing?

Experiment Instrumentation/set up

To meet above mentioned objectives, the multilayered hybrid regression testing technique (section 4.2) was selected for performing regression testing on embedded type software prototype payroll management software system. This Prototype software was delivered in TWO increments but was unsuccessful since the requirement prioritization process was faulty due to wrong decision aspect prioritization. About 20 students with different roles and views were selected for performing requirement prioritization and regression testing. Unsuccessful software was re-designed from scratch, also delivering it in TWO increments. The hybrid decision aspect technique as given in chapter 3 was employed for prioritization of decision aspects and for requirement prioritization, model as suggested by Gupta and Srivastav, (2011a), with slighter modification was executed. Two varieties of "hybrid regression testing technique" were executed on both unsuccessful (Project1) as well as successful projects (Project2). The known numbers of faults are seeded in both projects. In the first form of the technique (RT1), clustering on the basis of requirement priorities was ignored and all other steps were being executed on both projects. In another form of regression testing technique (with clustering on the basis of priorities) i.e. RT2, all steps of proposed technique were executed on both projects. In this chapter of the thesis, "Clustering" referred to both housing of test cases into clusters and cluster prioritization, which are the two different sequential steps of hybrid regression testing approach. The results of application of two varieties of same regression testing techniques on both projects were collected and analyzed against following parameters of interested i.e. "Change requests", "number of minimized test cases", "number of test cases", "Faults detected".

Results

After applying RT1 and RT2 on both projects, the values obtained against parameters of interest are given in Table 5.1 and Table 5.2.

Table. 5.1. Results of application of Regression Testing on Project1.

S.No

Parameters of Interest

RT1

RT2

Number of Test Cases (NT)

145+30*

145+30*

Reduced Test cases (RT)

84

51

Faults Detected (FD)

6/10

09/10

Change Requests (CR)

41 (16+25)**

41 (16+25)**

Table. 5.2. Results of application of Regression Testing on Project2.

S.No

Parameters of Interest

RT1

RT2

Number of Test Cases (NT)

122+30*

122+30*

Reduced Test cases (RT)

With clustering

Others Layers (Final

Reduction)

73

NA

73

32

54

32

Faults Detected (FD)

6/10

9/10

Change Requests (CR)

03 (2+1)***

03 (2+1)***

*a+b, a is total test cases, b is changed or newly added test cases, ** c+d, or ***c+d, c is CR for first increment, d is of next increment.

Analysis of Table 5.1 and 5.2, gives very interesting results. For unsuccessful Project1, although regression testing results in a good reduction in the number of test cases and fault detection rate were higher for RT2, yet the increased number of change request due to faulty decision aspect prioritization and hence faulty requirement prioritization made the whole testing effort as useless. Thus the effectiveness of regression testing does not matter if the requirement prioritization process is not accurate. Faulty requirement prioritization results in wastage of testing effort, since thorough testing of "Less favored" requirements makes no meaning to its stakeholders. For project2, clustering on the basis of requirement priorities i.e. RT2, resulted in testing of changed software with minimum test cases i.e. 32 (as compared to 73 of RT1), with the increased fault detection rate (6 to 9). This proved that if requirement priorities are taken as one of starting parameter during regression testing, one can achieve great reduction in testing effort. This also does not have any impact on fault detection rate.

Conclusion and Future work

Results show the direct impact of the requirement prioritization process and hence requirement priorities on regression testing activity. Effective requirement prioritization depends on the effectiveness of both decision aspect prioritization as well as requirement prioritization method. More effective the prioritization process is, the lesser is an amount of regression testing effort testing team has to invest.

Thus regression testing activity undertaken on evolving software is directly influenced by the requirement prioritization process as well as assigned priorities to individual requirements.

In future more regression testing approaches based on the requirements and their priorities are expected to emerge.

It is expected that the impact will further be investigated on different software of varying complexities by employing different requirement prioritization and regression testing techniques. The complexity in the techniques and software’s is one of the threats to validity.