A Parallel Tree Based Strategy

Published Date: 02 Nov 2017

Mohammad F. J. Klaib1, Sangeetha Muthuraman2, A.Noraziah 3

1Department of Software Engineering

Faculty of Sciences and Information Technology

Jadara University,

Irbid- Jordan

2,3 Faculty of Computer Systems and Software Engineering,

University Malaysia Pahang,

Pahang, Malaysia

Email: [email protected], [email protected]

Abstract. Software testing is the process which helps in identifying the correctness, completeness, and quality of developed systems. Lack of testing can lead to disastrous consequences including loss of data, fortunes, and even lives. Hence software testing is one of the most important phases of the software development life cycle. However, software testing is traditionally seen as a difficult and time consuming activity that is hard to embed in the software development process. Software and hardware testers concentrate on how to minimize the time involved in testing at the same time to ensure that the system is also tested well and made acceptable. Basic combinatorial interaction (i.e. pairwise or 2-way) testing has been one of the commonly used methods in achieving the above goal. Although 2-way interaction testing can detect 50-97 percent of errors, empirical evidence has proved that 2-way interaction testing is a poor strategy for testing highly interactive systems and it has been showed that most of the errors are triggered by the interaction of 2-6 input parameters. Therefore there is a need for higher T-way interaction testing to uncover these errors. Researchers have applied parallel algorithms to various large and/or computationally expensive optimization problems and have succeeded in solving these problems. Therefore, in this paper we have enhanced our previous strategy "A Tree Based Strategy for Test Data Generation and Cost Calculation for Pairwise Combinatorial Interaction Testing" to work effectively in parallel and to go beyond pairwise testing. The proposed strategy can now support a parallel 2-way and general T-way combinatorial interaction test data generation for uniform and non uniform parametric values. The proposed strategy is based on two algorithms; a parallel tree generation algorithm which generates the test cases and a parallel T-way cost calculation algorithm which is used in constructing test suites with minimum number of test cases. The correctness of the strategy has been proved, and the performance evaluation shows that our strategy is efficient in test size reduction.

Keywords: parallel algorithms, combinatorial interaction testing, software testing, hardware testing, T-way testing.

1. Introduction

To insure a high quality enhancement of software products, a good testing is required, where testing is the process of executing programs or systems with the goal of finding errors thereby assuring the correctness, completeness and quality of developed systems or computer software [5, 6, 41, 47]. Lack of testing can lead to disastrous consequences including loss of data, resources, and sometimes lives [10, 34]. Hence software testing has become an important and crucial phase of the software development life cycle [1, 2]. A well-tested product or service is necessity to ensure customer's satisfaction. However, exhaustive testing is unaffordable due to combinatorial explosion problem [24-27] which occurs when a small increase in the number of elements that can be combined increase the number of combinations to be computed.

How the problem of combinatorial explosion could be expensive and resource consuming? Consider for instance testing the IC7489 which is a 16 by 4 random access memory [46]. Pin assignment of inputs and outputs is shown in Figure 1. The four address inputs select one of the 16 words in the memory. The least significant bit of the address is A, and the most significant bit is C. The read/write control logic has two control inputs.  The memory enable (ME) input must be equal to low to enable the memory. If ME is high, the memory is disabled and all four outputs are at high impedance level.  The write enable (WE) input determines the type of operation as indicated in the function table (Table 1) that's shown below. The write operation is performed when WE is low.  This is a transfer of the binary data from the data input lines into the selected word in memory.  The read operation is performed when WE is high. This transfers the value stored in the selected word into the output data lines (complemented).

Fig. 1. IC7489, 16 X 4 random access memory

Table 1

Function table for 7489.

ME

WE

Operation

Data Outputs

L

L

Write

Complement of data inputs

L

H

Read

Complement of selected word

H

X

Disable

High impedance

In order to test this IC exhaustively there are 212 (i.e. 4096) combinations of test cases which are to be evaluated. If the time required for one test case to be evaluated is 2 minutes, then it would require nearly 17 working days for a complete test to be done. If 600 such IC’s needs to be tested then imagine the time and resources consumed for exhaustive testing to be done. Therefore, it is very clear that combinatorial explosion is a serious issue which has to be considered, and testing most hardware and software systems face this problem. Combinatorial explosion in testing may occur for configurable systems. When systems under test have many configuration parameters, each with several possible values, testing each configuration is sometimes infeasible.

Combinatorial interaction testing has been one of the methods used to minimize the size and the time involved in testing, at the same time to ensure that the system is also tested well and made acceptable. The combinatorial interaction testing approach can reduce the number of test cases by systematically selecting a subset from an exhaustive testing combination based on the strength of parameter interaction [17-22, 40]. Basic combinatorial interaction testing which is called pairwise or 2-way testing [12-16] provides a systematic approach to identify and isolate faults since many faults are caused by unexpected 2-way interactions among system factors. Empirical investigations have concluded that from 50 to 97 percent of software faults [4, 8, 14, 17, 23, 42, 43, 44] could be identified by pairwise combinatorial interaction testing. However, what about the remaining faults? Especially, in case of highly interactive systems which have a number of interactions with higher strength. How many failures could be triggered only by an unusual interaction involving more than two parameters? Investigations have found that many faults were caused by a single parameter, a smaller proportion resulted from an interaction between two parameter values, and progressively fewer were triggered by 3-6 way interactions [11, 36, 37, 38, 34, 35].

Therefore, to ensure a high quality testing of complex applications, it is necessary to generate test suites for higher degree T-way interactions. T-way testing [11, 36, 37, 38, 34, 35] requires every combination of any T parameter values to be covered by at least one test, where T is referred to as the strength of coverage. If all faults in a system can be triggered by a combination of T or fewer parameters, then testing all T-way combinations of parameters can provide high confidence that nearly all faults have been discovered. A number of studies have shown combinatorial methods to be highly effective for software and hardware testing [4, 8, 11, 14, 17, 23, 37, 38, 42, 43, 44, 34, 35].

Large and/or computationally expensive optimization problems sometimes require parallel or high-performance computing systems. Parallel algorithms have been applied to problems such as weather and climate modelling, bioinformatics analysis, logistics and transportation, and engineering design. Furthermore, commercial applications are driving development of effective parallel software [28, 29, 38, 44] for large-scale applications such as data mining and computational medicine. In the simplest sense, parallel computing involves the simultaneous use of multiple computer resources to solve a computational problem.

In this paper we have enhanced our previous strategy "A Tree Based Strategy for Test Data Generation and Cost Calculation" [42, 43, 44] to work in parallel and to go beyond pairwise (2-way) testing. The proposed strategy can now support a parallel and general T-way combinatorial test data generation involving uniform and non uniform parametric values. The proposed strategy is based on two algorithms; a parallel tree based test data generation algorithm which generates all the test cases, and a parallel T-way cost calculation algorithm which is applied to construct T-way test suites with minimum number of test cases. The remainder of this paper is organized as follows. Section 2 presents the related work. Section 3 explains the parallel tree generation and the proposed iterative T-way cost calculation strategy with an example. Section 4 gives parallel tree generation algorithms for test case generation and explains its advantages. Section 5 presents the parallel, iterative, T-way cost calculation algorithm for T-way test suites generation, with its working explained. In Section 6 the performance of the proposed strategy has been evaluated and finally, Section 7 gives the conclusion and the future enhancements that could be made.

2. Related Work

There are a number of strategies proposed in literature for test data generation based on combinatorial interaction testing. Some of these strategies work only for pairwise combinatorial interaction testing and some others have been extended to work for T-way testing. These strategies could be broadly classified into two types [12] based on the approach that is used to solve the problem. They are:

Algebraic Strategies

Computational Strategies

Algebraic approaches have pre-defined rules to compute test suites directly from mathematical functions [12, 3]. On a contrary, Computational approaches are based on the generation of all T-way combinations. Then it uses an iterative search technique and searches the combinations space to generate the test cases until all T-way combinations of interactions have been covered. A number of researches have worked in this field and have adopted either the computational or algebraic approach.

The classification of strategies used for combinatorial interaction testing has been further extended by Grindal et al. [24] again into three main categories based on the randomness of the implemented solution. They are:

Deterministic Strategies

Non-Deterministic Strategies

Compound Strategies

A Deterministic strategy is one which has the property that it produces the same test suite for every execution. A non-Deterministic strategy on the other hand has the property that for every execution, there is always a randomly generated combination suite to cover all the required T-way combinations. In a Compound strategy two or more combination of strategies are used together [24].

The Automatic Efficient Test Generator or AETG [8, 13] and its variant mAETG [9, 12] employ the computational approach. This approach uses ‘Greedy technique’ to construct test cases based on the criteria that every test case covers as many uncovered combinations as possible. The AETG uses a random search algorithm and hence the test cases are generated in a highly non-deterministic fashion [27]. Other variants of AETG use the Genetic Algorithm, Ant Colony Algorithm [15].

In Genetic algorithm [15] an initial population of individuals (test cases) are created and then the fitness of the created individuals is calculated. Then the individual selection methods are applied to discard the unfit individuals. The genetic operators such as crossover and mutation are applied to the selected individuals and this continues until it evolves a set of best individuals or the stopping criteria is attained. Thus this approach follows a non deterministic methodology similar to the Ant Colony Algorithm [15] in which each path from start to end point is associated with a candidate solution. The candidate solution is the amount of pheromone deposited on each edge of the path followed by an ant, when it reaches the end point. When an ant has to choose among the different edges, it would choose the edge with a large amount of pheromone with higher probability thus leading to better results. In some cases, these algorithms give optimal solution than original AETG.

The In-Parameter-Order [14] or IPO Strategy for pairwise testing uses an algebraic approach. It starts constructing the test cases by considering the first two parameters, then uses a horizontal growth strategy which extends to cover the third, fourth, fifth etc. until all the parameters are considered. Further it adopts a vertical growth strategy which helps in covering all the uncovered pairs, until all the pairs in the covering array are covered. Thus this approach generates the test cases in a deterministic fashion. Covering one parameter at a time gives a lower order of complexity to this strategy than AETG. The IPOG [11] works in the same way as that of IPO; But IPOG can generate test suite supporting T-way combinatorial interactions. The IRPS Strategy [45] uses the computational approach and so generates all pairs and stores them in a linked list and then searches the list to arrive at the best set of test cases in a deterministic fashion.

The G2Way [4, 7] uses a computational and deterministic approach. It adopts a backtracking strategy to generate the test cases. The main algorithms that form the G2Way strategy consist of the parser algorithm, the T-way combination generation algorithm, the backtracking algorithm, and the executor algorithm. The parser algorithm will load the parameter and values to be used by the T-way combination generation algorithm which generates the T-way covering array. Exploiting the result generated by the combination generation algorithm, the backtracking algorithm generates the T-way test sets in two phases. In the first phase, the sets generated by the combination generation algorithm are merged together to form complete test suites. In the second phase, all the test sets in the generated test suite are checked to ensure that all the combinations in the covering array are covered. Finally, upon completion, the backtracking algorithm forwards the results to the executor algorithm for execution. GTWay adopts the same strategies as that of G2Way but generates test suites for T-way combinatorial interaction [34].

The TConfig [31] uses a deterministic approach to construct test suites for T-way testing. It uses a recursive algorithm for pairwise interaction testing and a version of IPO for T-way testing. Tconfig was mainly developed for pairwise interaction test suite generation by applying the theory of Orthogonal Latin Squares from balanced statistical experiments. Jenny [32] is a tool similar to AETG, which first covers single features (one way interaction), then pairs (2-way interaction) of features, then triples (3-way interaction), and so forth up to the n-tuples requested by the user. During each pass it checks whether the existing tests cover all tuples, and if not, make a list of uncovered tuples and add more tests until all tuples are covered. It tries to find test cases that obey the restrictions and cover a lot of new tuples. Any tuple that it can't cover no matter how hard it tries without disobeying some restriction, it says it can't cover it and adds it to its list of restrictions. Thus it uses a computational and deterministic approach for test suite generation.

WHITCH is IBM’s Intelligent Test Case Handler. With the given coverage properties it uses combinatorial algorithms to construct test suites over large parameter spaces. TVG [33] is a free tool that is built based on model based techniques. It combines both behavior and data modeling techniques. The behavior modeling allows the testers to capture important high level scenarios to test. A data model is created at a level of sophistication according to the importance of each test scenario.

Other researchers have adopted heuristic search techniques [24] such as Hill climbing, Simulated Annealing, Tabu search, Great Flood etc. All of these search strategies have the same goal as to maximize the number of t- tuples covered in a test. It initially uses greedy algorithm to choose each test and then it is modified using local search. These Heuristic search techniques predict the known test set in advance in contrast to AETG and IPO which builds the test set from the scratch. However, there is no guarantee that the test set produced by Heuristic techniques are the most optimum.

Few researchers worked on test suite generation for combinatorial interaction testing using parallel, distributed and grid computing based strategies. Multicore modified input parameter order (MC-MIPOG) [38] is an extension of modified IPOG which executes in a parallel fashion. This strategy distributes the computational process and memory into pieces. Calvagna et. al. [39] proposed a new strategy for T-way test suite construction using the grid computing environment. Therefore not much work has been done in T-way testing. Combinatorial interaction testing is a NP complete problem that needs exploration with efficient parallel techniques to reduce the generation cost.

3. The Proposed Strategy

The proposed strategy constructs the tree based on the parameters and values given. It constructs every branch of the tree in parallel. The number of branches the tree has depends on the number of values of the first parameter i.e. if the first parameter has 3 values then the tree also would have 3 branches. Therefore every branch construction starts by getting one value of the first parameter i.e. branch T1 gets the first value, T2 gets the second value and so on. After the base branches are constructed one child thread is assigned to every branch and the further construction takes place in a parallel manner. Each of the branches considers all values of all the other parameters two, three,…..N where N is the total number of parameters. All the branches consider the values of the parameters in the same order. The following simple system with parameters and values, illustrates the concept as shown below:

Parameter A has two values A1 and A2

Parameter B has one value B1

Parameter C has three values C1, C2 and C3

Parameter D has two values D1 and D2

We have given the illustration for minimum test suite construction of 2-way and 3-way combinatorial interactions testing using our algorithm, for the system mentioned. The algorithm starts constructing the test-tree by considering the first parameter. As the first parameter has two values the tree is said to have two main branches with the first branch using A1 and the second branch using A2. Then each of the branches is constructed in parallel by considering all the values of the second parameter, then the third and fourth and so on. When the branches are fully constructed the leaf nodes gives all the test cases that has to be considered for cost calculation. Since all of the branches are constructed in parallel there is a significant reduction in time. Fig. 2 shows the test tree for the system above.

Fig. 2. Test-tree construction

Fig. 2 above shows how the test-tree would be constructed. The test cases generated by the first branch are stored in the lists T1 and the test cases generated by the second branch are stored in T2 respectively. i.e. (A1,B1,C1,D1), (A1,B1,C1,D2), (A1,B1,C2,D1), (A1,B1,C2,D2), (A1,B1,C3,D1), (A1,B1,C3,D2) are stored in T1, and (A2,B1,C1,D1), (A2,B1,C1,D2), (A2,B1,C2,D1), (A2,B1,C2,D2), (A2,B1,C3,D1) and (A2,B1,C3,D2) are stored in T2.

Once the parallel tree construction is over we are ready with all the test cases to start the parallel iterative cost calculation. In this strategy the cost of the leaf nodes in each of the lists are calculated in parallel in order to reduce the execution time. The cost of a particular test case is the maximum number of T-way combinations that it can cover from the covering array. At First, the algorithm starts by constructing the covering array, for all possible T-way combinations of input variables, if T equals 2 i.e. [A & B], [A & C], [A & D], [B & C], [B & D] and [C & D]. The covering array for the above example has 23 pairwise interactions as shown in Table 2, which has to be covered by any test suite generated, to enable a complete pairwise interaction testing of the system.

Once the covering array is generated the algorithm starts to include all tree branches. which might definitely give the maximum Wmax cost into the test suite. Then these test cases are deleted from the tree branches lists T1 and T2, and the corresponding pairs covered by it in the covering array are also deleted. In the third step, the main thread in the algorithm invokes a number of child threads equal to the number of values of the first parameter and calculates the cost of all the test cases in each of the branches in a parallel fashion. Each child thread stores all the test cases with the Wmax value from its corresponding branch into a separate sub-list. The child thread that finishes calculating the cost of all the test cases in its branch first locks the covering array. This thread then looks into its sub-list and includes the test cases stored in it into the test suite only after confirming that the test case definitely has the maximum cost or Wmax value. Then the test cases included in the test suite are deleted from the tree branches list and sub-list, and the corresponding pairs that these cover are deleted from the covering array.

Table 2

Pairwise covering array.

A with B

A with C

A with D

B with C

B with D

C with D

A1,B1

A1,C1

A1, D1

B1,C1

B1, D1

C1, D1

A2,B1

A1,C2

A1, D2

B1,C2

B1, D2

C1, D2

A1,C3

A2, D1

B1,C3

C2, D1

A2,C1

A2, D2

C2, D2

A2,C2

C3, D1

A2,C3

C3, D2

The other threads wait in a queue until the execution of the first thread is over, after which these threads resume their execution in the order in which they are queued. These threads on resumption re-evaluate the test cases in their sub-list to confirm that these test cases have the Wmax value before including these into the test suite. Thus in the first iteration all the test cases with the maximum Wmax value from all the branches are included in the test suite. Now the Wmax value is decremented by one and the same parallel execution of all the threads continue until all the pairs in the covering array are covered. For the above example all the test cases which are included in the test suite are identified in four iterations and there are six such test cases. Table 3 shows how the cost calculation works iteratively to generate the test suite. The same test suite gets generated if a sequential execution of the above algorithm takes place.

Table 3

Generated test suite for pairwise combinatorial interaction.

Test Case No.

Test Case

Iteration/Child Thread No.

Max

Weight

Covered pairs

T1

A1,B1,C1,D1

1/1

6

[A1,B1][A1,C1][A1,D1]

[B1,C1][B1,D1][C1,D1]

T10

A2,B1,C2,D2

1/2

6

[A2,B1][A2,C2][A2,D2]

[B1,C2][B1,D2][C2,D2]

T6

A1,B1,C3,D2

2/1

4

[A1,C3][A1,D2]

[B1,C3][C3,D2]

T11

A2,B1,C3,D1

3/2

3

[A2,C3] [A2,D1] [C3,D1]

T3

A1,B1,C2,D1

4//1

2

[A1,C2] [C2,D1]

T8

A2,B1,C1,D2

4/2

2

[A2,C1] [C1,D2]

As the pairwise test suite is generated, we can generate the test suite for 3-way combinatorial interactions and so on the forth until (n-1) way combinatorial interaction test suites are generated. To illustrate the 3-way test suite generation, again the whole process starts by constructing the 3-way covering array and the iterative, parallel cost calculation of the test cases in the various branches as explained before. Table 4 shows the covering array for 3-way combination i.e. [A, B, C], [A, B, D], [A, C, D] and [B, C, D], for the example in Fig. 2. The covering array for the above example has 28 3-way interactions which have to be covered by any test suite generated, to enable a complete 3-way interaction testing of the system. Table 5 shows how the cost calculation works iteratively to generate the test suite. Table 5 also shows the order in which the various test cases are actually included in the test suite.

Table 4

3-way interaction covering array.

A, B, C

A, B, D

A, C, D

B, C, D

A1, B1, C1

A1, B1, D1

A1, C1, D1

B1,C1, D1

A1, B1,C2

A1, B1,D2

A1, C1, D2

B1,C1, D2

A1, B1,C3

A2, B1,D1

A1, C2, D1

B1,C2, D1

A2, B1,C1

A2, B1,D2

A1, C2, D2

B1,C2, D2

A2, B1,C2

A1, C3, D1

B1,C3, D1

A2, B1,C3

A1, C3, D2

B1,C3, D2

A2, C1, D1

A2, C1, D2

A2, C2, D1

A2, C2, D2

A2, C3, D1

A2, C3, D2

Table 5

Generated test suite for 3-way combinatorial interaction.

Test Case No.

Test Case

Iteration/ Child Thread No.

Max

Weight

Covered pairs

T1

A1,B1,C1,D1

1/1

4

[A1,B1,C1][A1,B1,D1][A1,C1,D1][B1,C1,D1]

T4

A1,B1,C2,D2

1/1

4

[A1,B1,C2][A1,B1,D2][A1,C2,D2][B1,C2,D2]

T8

A2,B1,C1,D2

1/2

4

[A2,B1,C1][A2,B1,D2][A2,C1,D2][B1,C1,D2]

T9

A2,B1,C2,D1

1/2

4

[A2,B1,C2][A2,B1,D1][A2,C2,D1][B1,C2,D1]

T5

A1,B1,C3,D1

2/1

3

[A1,B1,C3][A1,C3,D1][B1,C3,D1]

T12

A2,B1,C3,D2

2/1

3

[A2,B1,C3][A2,C3,D2][B1,C3,D2]

T2

A1,B1,C1,D2

3/1

1

[A1,C1,D2]

T3

A1,B1,C2,D1

3/1

1

[A1,C2,D1]

T6

A1,B1,C3,D2

3/1

1

[A1,C3,D2]

T7

A2,B1,C1,D1

3/2

1

[A2,C1,D1]

T10

A2,B1,C2,D2

3/2

1

[A2,C2,D2]

T11

A2,B1,C3,D1

3/2

1

[A2,C3,D1]

4. Parallel Tree Generation Algorithms for Test Case Generation

4.1 Tree Generation Algorithm for Main Thread

Input: A set of parameters and the values of the corresponding parameters

Output: Lists of test cases. Each list holds the Test cases generated by the tree in one particular branch of that tree.

Begin

X = number of values of first parameter p1

{For the first parameter p1}

Ti=Vi, where i=1,2,3,……..,X/ parameter p1 has X values

If N=1 then stop and exit;

Create X threads with unique thread ids. Assign each Ti to a separate child thread and execute all the child threads in parallel

Wait for the termination of all the threads to get the results from all the branches.

End

4.2 Tree generation Algorithm for Child Thread

Begin

{For the remaining parameters the execution takes place in parallel}

For parameters Pj, j=2,3,………N do

Where N is the total number of parameters

Begin

For each Test (Vi1, Vi2,………….Vim) in Ti do

Where i = 1,2,…..X, X is the number of values of parameter p1and m is the maximum number of test cases in list Ti at that Time

Begin

Replicate the Test as many times as (the number of values of Pj – 1)

Add all the replicated nodes sequentially after the current original test node and before the other test nodes in Ti

For each value in Pj do

Begin

Append the original node with V1 and all the replicated tests with (V2, V3,……..Vy-1, Vy) where Vy is a value of Pj and each of which is considered in order.

End

End

End

End

The tree generation algorithm thus provides the following advantages:

A systematic method whereby all possible test cases are generated in order.

The above procedure works fine with the parameters having any number of values. Therefore all parameters can have different or same values as any real time system to be tested would have.

The procedure appears to generate the full tree by using all the values of the parameters but at every iteration only a set of leaf nodes are left thus having a list of leaf nodes ( or test cases) when the procedure ends.

Since the test cases in every branch are generated in parallel by the child threads there is significant reduction in time.

The example tree shown in Fig. 2 explains how the test cases are constructed manually. In reality we may need only the leaf nodes and all the intermediate nodes are not used. Therefore in order to increase the efficiency of the implementation we have constructed the same tree as in Fig. 2 using the proposed parallel tree generation algorithm. This proposed algorithm constructs the tree by minimising the number of nodes. Minimisation of the number of nodes is achieved by giving importance only to the leaf nodes at every stage.

The main thread just constructs the base branches of the tree each of which consists of one value of the first parameter in an order in which the input was made. Therefore, in the example above there are only two base branches and the value A1 is assigned to branch T1 and A2 to T2. Then the main algorithm invokes a number of unique child threads to handle each of the branches separately. At each stage or iteration each of the child threads look at the leaf nodes of their corresponding branches and generate the next level nodes by considering all the values of the current parameter, to generate the new set of nodes. The new set of leaf nodes from an already existing set is calculated using a replication strategy. The existing set of leaf nodes be Esoln, new set of leaf nodes be Nsoln and the number of values of the parameter under consideration be n. Then,

Nsoln = Esoln * n

(0)

Let there be 4 leaf nodes in a branch and the next parameter to be considered has 2 values. Then the new list of nodes for that branch will have 8 new leaf nodes as a result. The algorithm considers every leaf node separately and calculates the number of times this particular node needs to be replicated with the formulae given below:

The number of values of pj – 1

(2)

Where pj – is the jth parameter under consideration for constructing the new set of leaf nodes and j=1, 2,….N – the number of parameters. In the Fig. 2 that is shown above consider the leaf nodes (A1, B1) of list or branch T1 and (A2, B1) of branch T2. To construct the next level of leaf nodes the parameter under consideration is C, which has values C1, C2 and C3. Therefore, the node (A1, B1) needs to be replicated twice. Now we will have three (A1, B1) nodes to which C1 is added to the first, C2 is added to the second and C3 is added to the third and then the replicated nodes are included in the list of leaf nodes after the original node. The same is done to (A2, B1). It is replicated twice and hence we have three of it (one original and two replicated nodes). Now C1 is added to the first (original node), C2 is added to the second (replicated node) and C3 is added to the third (replicated node). Thus we have (A2, B1, C1), (A2, B1, C2) and (A2, B1, C3). If there are more parameters the same is continued until all the parameters are considered. Thus, once the lists of leaf nodes are generated we go to the next strategy of iterative and parallel cost calculation to construct the test suite.

5. Test Suite Generation by Iterative and Parallel Cost Calculation Strategy

5.1 Strategy T-way Test Suite Generation by Iterative and Parallel Cost Calculation (Main Thread)

Input: Lists of test cases. Each list holds the test cases generated by the tree in one particular branch of that tree.

Output: T-way test suites with minimum number of test cases

Begin

Tempb = Tb (where b is the number of lists of test cases)

X = number of values of parameter p1

B=min (Value(p1), Value(p2), …….Value(pn)

For i = 2 to N-1 do

Begin

Generate the i-way covering array for the given parameters.

Wmax = N!/((i!)*((N-i)!)) // N – is the number of parameters

Let T’ be an empty set where i-way test suites are stored.

For a = 1 to B do

Begin

Testa = concatenate the ath values of all the parameters to form a test case.

End

For each Testa do

Begin

Delete all the T-way combinations that Testa covers in the covering array

Delete Testa from the Ti Lists

T’ = Testa

End

Creates a set of temporary lists Yi corresponding to the Ti lists, where i= 1,2,…..X, X is the number of values of parameter p1 or the number of lists

Create X threads with unique thread ids. Assign every child thread Thi with one Ti lists, the corresponding Yi lists, i value and Wmax value, and execute all the child threads in parallel

Wait for the termination of all the child threads.

Store the i-way test suite generated in the list T’

Tb = Tempb

End

End

5.2 Strategy T-way Test Suite Generation by Iterative and Parallel Cost Calculation (Child Thread)

Begin

While (covering array is not empty) do

Begin

For each Test Tij in Ti do

Where i=1,2,…….X, X – is the number of lists and j=1,2,….n where there are n test cases in Ti at that time

Begin

Cost[Tij]= The number of T-way combinations covered by it in the covering array

If (Cost[Tij]==Wmax)

Begin

Yi = Tij

End

End

{Whichever thread completes its execution first locks the covering array and updates all its test cases with Wmax values from Yi to the Test suite T’ and deletes all the corresponding T-way combinations of those test cases included in T’ from the covering array. The other threads on completing execution enters a queue and does its updation in that queued order by locking and unlocking the covering array after the first thread releases its lock on the covering array }

For each Yi do (lock the covering array and make updation)

Begin

If (Yi != empty)

Begin

For each Test Tij in Yi do

Begin

Count= The number of T-way combinations covered by it in the covering array

If (Count ==Wmax)

Begin

T’ = T’ U Tij

Delete all the T-way combinations that Tij covers in the covering array

Delete Tij from the lists Ti

End

Delete Tij from the lists Yi

End

End

(unlock the covering array)

End

Wait until all child threads finishes updating

Wmax=Wmax-1

End

End

The main thread includes the base test cases which would definitely have a maximum cost value and then invokes a number of unique child threads which operate in parallel on each of the branches lists. The main thread iterates N-2 times thus generating N-2 test suites. In the first iteration, i=2, the child threads iterate through the lists of test cases until all the pairs of the 2-way covering array are covered. Then the minimum 2-way test suite generated is stored and the next iteration begins. Now, i=3 and the child threads iterates again through the lists of test cases until all the 3-way combinations of the 3-way covering array are covered and then the 3-way test suite generated is stored. Thus this is continued until i= N-1. At each iteration, all the test cases with the maximum cost (Wmax) for that particular iteration are included in the test suite. Thus the algorithm guarantees identifying minimum test suites for parameters with same as well as different number of values.

6. Performance Evaluation

The correctness of the proposed strategy TBGCC has been proved in Section 3 with an example as shown in Fig. 2. For this example the generated 2-way and 3-way covering arrays has been shown in Table2 and Table4 respectively. The test suite generated by our system has covered all the pairs in Table 2 as shown in Table 3, and all 3-way combinations in Table 4 as shown in Table 5, thus proving the correctness of the strategy.

To evaluate the efficiency of the strategy for T -way test data generation, we consider six different configurations. The first three configurations have non-uniform parametric values. The other three configurations have a uniform number of values for all parameters. The six system configurations used are summarized as follows:

S1: 3 parameters with 3, 2 and 3 values respectively.

S2: 4 parameters with 2,1,2 and 1 values respectively

S3: 5 parameters with 3, 2, 1, 2 and 2 values respectively.

S4: 3 3-valued parameters

S5: 4 3-valued parameters.

S6: 5 2-valued parameters.

In Tables 6, 7 and 8, the column 2 shows the exhaustive number of test cases for each system. Column 3 displays the generated test suite size for 2-way, 3-way and 4-way interactions using our algorithm. The last column shows the percentage of reduction achieved by using our strategy.

The results in the Tables 6, 7 and 8 demonstrate that our strategy is an efficient strategy in test size reduction. If we consider the table 6 with pairwise test suite size and reduction, in some cases a high reduction is achieved, as in systems S5 and S6. In case of system S2, there is no reduction (0%) achieved because this is the minimum test suite size. In case of table 7 which shows the 3-way results there is reduction achieved in case of systems S3, S5 and S6, but in case of S2 no reduction is achieved as this is the minimum test suite size. The other systems such as S1 and S4 have 3 parameters only and therefore cannot be considered for 3-way test suite reductions. Similarly, in case of table 8 which shows the 4-way results, 46.88% of reduction is achieved in case of S6 and no reduction is achieved in case of S3. The other systems have only 4 parameters and hence cannot be considered for 4-way test suite generation. Thus, the tables 6, 7 and 8 reveal that the proposed strategy works well for T-way test suite size reduction, for both parameters with uniform as well as non-uniform values.

Table 6

TBGCC 2-way results.

System

Exhaustive number of test cases

TBGCC

2-way Test suite size

2-way Reduction %

S1

18

9

50%

S2

4

4

0%

S3

24

7

70.83%

S4

27

9

66.67%

S5

81

9

88.89%

S6

32

7

78.13%

Table 7

TBGCC 3-way results.

System

Exhaustive number of test cases

TBGCC

3-way Test suite size

3-way Reduction %

S2

4

4

0%

S3

24

16

33.33%

S5

81

31

61.73%

S6

32

12

62.5%

Table 8

TBGCC 4-way results.

System

Exhaustive number of test cases

TBGCC

4 -way Test suite size

4 -way Reduction %

S3

24

24

0%

S6

32

17

46.88%

In the reminder of this section, we will study the efficiency of our strategy in test size reduction by comparing our results with available testing strategies which is also based on parametric interactions. We have identified the following existing strategies that support T-way testing: AETG [8,13] , IPO[14], AllPairs [30], TConfig [31], Jenny [32], TVG [33], G2Way[4], GTWay [34] tool. We consider the same six system configurations as described above.

Tables 9 and 10 displays the comparison of test suite size generated by our strategy with other strategies. The minimum test suite size is highlighted. Our strategy (TBGCC) has produced the best results for some configurations and in some other cases the other systems have performed better than us. However, test reduction is a NP-complete problem [15, 16], for which, no strategy may perform the best for all cases.

Table 9

Comparison of TBGCC with other strategies for generated 2-way test suite size.

System

AETG

IPO

TConfig

Jenny

TVG

ALL Pairs

G2Way

PTBGCC

S1

NA

9

9

9

10

9

9

9

S2

NA

NA

NS

NS

4

4

4

4

S3

NA

7

NS

NS

8

9

7

7

S4

NA

10

9

9

9

10

10

9

S5

9

10

10

13

13

10

10

9

S6

NA

NA

6

7

6

6

6

7

Table 10

Comparison of TBGCC with other strategies for generated 3-way test suite size.

System

TConfig

Jenny

TVG

GTWay

PTBGCC

S2

NS

NS

4

4

4

S3

NS

NS

13

12

16

S5

32

34

33

32

31

S6

12

12

12

12

12

NA – Not Available and NS – Not Supported

7. Conclusion and Future Work

In this paper we have proposed a parallel tree based test data generation and parallel iterative cost calculation strategy for T-way combinatorial interaction testing. The correctness of the proposed strategy has been proved in section 3 (Tables 2, 3, 4 and 5). The performance of the strategy has been evaluated in Section 5 which demonstrates that our strategy is an efficient strategy in test size reduction and can generate highly reduced test suites. The proposed strategy is different from other strategies because at every iteration only the test cases which have covered the maximum number of T-way combinations have been included into the test suite. The algorithms proposed could be extended in future to work for different models of shared and distributed memory architectures.