The Need For Reliable System Software

Published Date: 02 Nov 2017

CHAPTER 1

INTRODUCTION

Software has become a vital component of our modern society such as home appliances, telecommunications, automobiles, airplanes, shopping, web teaching, personal entertainment, and so on. All such applications require correct and reliable software for providing desired services and functionality. This way, prediction and assessment of reliability has become increasingly important due to the impact of software failures and cost of development for reducing the failures. Measurement of reliability is an extremely challenging area due to non-availability of universally acceptable metrics for the assessment and prediction of various characteristics of software systems.

1.1 What is Reliability?

Reliability is defined as the probability that a component or system will perform the required function for a given period of time when used under stated operating conditions (Ebeling, 2007). It is the probability of a non failure over time. In its simplest form, reliability means the probability that a failure may not occur in a given time interval. Thus, reliability definition stresses upon four components like probability, intended function(s), time, and operating conditions. Reliability also depends on environmental conditions which may or may not vary with time. Since reliability is defined in terms of probability, its numerical value is always between one and zero.

Reliability Modeling

Reliability is increasingly becoming more important during the design of software systems, as our daily lives and schedules are more dependent than ever before on the satisfactory functioning of these systems. Reliability is the most quality attribute of any software. That is why modeling reliability of software systems has become more important than ever before. Although, reliability can be grown through various means like improving the process of designing, effectiveness of testing, manual and automated inspections, familiarization with developers, and improving the man­agement processes (Musa, 1984).

The rate at which reliability grows depends on several factors like how rapidly defects are observed, how fast corrective action can be iden­tified and how soon the impact of changes takes place in the operational field. Software reliability modeling has been around since early 1970s and the basic approach is to model past failure data to predict future behavior of software (Moranda, 1971). This approach employs either the observed number of failures discovered per time period or observed time between failures of software (Musa, 1987).

Thus, software reliability models aim at predicting critical software modules/components prior to testing. These models are generally built from metrics collected from the past projects or software releases which are used to identify fault prone modules/components in the current software systems (Musa, 1975; Littlewood, 1979; Basili, 1981).

1.3 The Need for Reliable System Software

The size and complexity of software systems have grown dramatically during the past few decades, and the trend will certainly continue in the future. The data from industry shows that the applications of software for various systems have been growing exponentially for the past forty years (Jelinski and Moranda, 1972; Goel and Okumoto, 1979; Littlewood, 1979; Musa, 1980; Basili, 1981; Yamada et al., 1984; Pham 2006). Thus, we need to improve the reliability of software systems due to various reasons discussed as follows:

Systems are becoming software oriented day by day and many software intensive systems are safety critical.

Software users are demanding reliable and warranted software systems.

The cost of software development is increasing very rapidly.

Increasing competition and high development costs have intensified the pressures to quantify software quality as per the local needs and global standards.

The measurement and quality control of software systems is resource consuming and complex task.

1.4 Software Reliability

Software reliability is the most important and most measurable aspect of software quality. It is a measure of how well the program functions to meet its operational requirements. Some of the widely used and accepted definitions of software reliability in practice are discussed as follows:

1.4.1 Definitions

The most acceptable definition of software reliability is "the probability of a failure free operation of a program for a specified period in a specified environment" (Musa, 1987).

According to Littlewood, "Software reliability means operational reliability: we should be concerned with their effect on its operations" (Littlewood, 1979).

The IEEE defines reliability as "the ability of a system or component to perform its required functions under stated conditions for a specified period of time".

The NASA software assurance standard, NASA-STD-8739.8, defines software reliability as a discipline of software assurance composed of three components: (i) to define the requirements for software controlled system failure detection, isolation, and recovery (ii) reviews the software development processes and products for software error prevention and (iii) defines the process for measuring and analyzing defects and defines the reliability and maintainability factors.

Software reliability is also defined as the probability that a software system fulfills its assigned task in a given environment for a predefined number of input cases, assuming that the hardware and the inputs are free of error (Kope, 1979).

Thus, software reliability is the probability that software will work without failure for a specified period of time in a given environment. The environment and time is fixed. That is, reliability is for fixed time under given environment or stated conditions.

Mathematically, we can express reliability, R(t) as the probability that a system will be successful in the interval from time 0 to time t which can be expressed as (Musa, 1987; Pham, 2006):

(1.1)

where T is a random variable denotes the time-to-failure or failure time and the reliability is a number between 0 and 1.

1.4.2 Software Reliability Curve

Typically there are three phases in the life of any hardware component like burn-in phase, useful life and wear-out phase shown as in figure1.1. During burn-in phase the failure rate is quite high initially, and it starts decreasing as the testing time progresses. It may be due to initial testing in the premises of the organization. In useful life period, the failure rate is approximately constant. Further, failure rate increases during wear-out phase due to wearing out or aging factor of the components.

Figure1.1 Software reliability curve: failure rate versus time

The best period is useful life period in a typical bath-tub curve of hardware reliability. We do not have such wear out phase in software. The software may be retired only if it becomes obsolete. Some of the significant factors which affect the useful life of software are described as follows (Musa, 1987; Pham, 2006; Aggarwal and Singh, 2008):

Change in environment/ customer requirements changes

Change in infrastructure or technology changes/increase in complexity

Difficult to maintain due to high complexity or maintenance cost becomes quite high

Deterioration in structure of the source code or due to poor/ slow execution speed

Poor graphical user interfaces or difficult to handle extremely large size databases

The expected software reliability curve is shown in figure1.1. In practice, the software failure rate is at the highest level during integration and testing. As the testing proceeds, the errors are identified and removed. However, the error removal continues at a slower rate during its operational use, but the number of errors continually decreases assuming no new faults/errors are introduced during debugging. Software does not have any moving parts and does not physically wear out as hardware, but it does outlive its usefulness and becomes obsolete.

1.4.3 Failure Intensity

The failure intensity (λ) is defined as the number of units failing per unit time. The failure rate changes throughout the life of the software product that provides us the familiar bathtub curve. The bathtub curve describes the failure rate phenomena consists of three parts where the first part is a decreasing failure rate, known as early failures. The second part is a constant failure rate, known as random failures and the third part is an increasing failure rate, known as wear-out failures (Musa, 2007).

The failure behavior is affected by two main factors namely the number of faults in the software being executed and the execution environment (or the operational profile of execution). The number of faults in the software is the difference between the number of faults introduced and number of faults removed. The faults are introduced by the programmer during coding or during original design or when the new features are added/ making design changes or repairing faults that have been identified.

In general, only source code that is new or modified results in faults introduction. The coding part that is inherited from another application does not usually introduce any appreciable number of faults except possibly during the interfaces. Since, it has been thoroughly debugged in the previous application. The process of fault removal introduces some new faults because it involves modification or writing of new source codes for feature enhancement.

1.5 Hardware Reliability versus Software Reliability

Although, probabilistic definitions of hardware and software reliability are identical but make huge difference between the two. Hardware reliability of a product is defined as the probability that the device will perform its intended function for at least a specified period of time under specified environmental conditions. Predicting software reliability is a different concept as compared to predicting hardware reliability.

Software becomes more reliable over time, instead of wearing out. Software becomes obsolete as the environment for which it was developed changes. Hardware redundancy allows us to make a system reliable as we desire, if we use large number of components with given reliability (Yamada et al., 1986; Xie, 1991; Lyu, 2005; Aggarwal and Singh, 2007). But, we do not have such mechanism in foreseeable future of software.

Software Reliability Modeling Techniques

Software reliability modeling is a mechanism which can be used to evaluate the software quantitatively, provide development status, schedule status and monitor the changes in reliability performance. Most of the software reliability models therefore fall into two basic classes, depending upon the types of data the model uses: failures (fault) count and time between failures (Boehm 1989; Cai et al., 1991; Wood, 1996; Pham, 2003).

The main factors which affect software reliability modeling includes fault introduction, fault removal and the environment. Fault introduction depends primarily on the characteristics of the developed source code (code created or modified for the application) and development process characteristics such as software tools, techniques, technologies and the developer’s experience. Fault removal depends upon time, operational profile and the quality of repair/correction activities. The environmental factors mainly depend on the operational profile. These factors are probabilistic in nature and operate over time that is why software reliability models are generally formulated in terms of the random processes.

Broadly, software reliability modeling techniques can be categorized as trending reliability and predictive reliability. The trending reliability tracks the failure data produced by the software system to develop a reliability operational profile of the system over a specified time. The predictive reliability assigns probability to the operational profile of software system (Yamada et al., 1984; Musa, 1998; Kai et al., 2001; Pham, 2006; Kue et al., 2008). Furthermore, trending reliability can be classified into four sub-categories such as error seeding, failure rate, curve fitting, and reliability growth discussed as below.

Error Seeding

This estimates the number of errors in a program by using multistage sampling.  The errors are divided into indigenous and seeded errors.  The unknown number of indigenous errors is estimated from the number of induced errors and the ratio of errors obtained from debugging data.

Failure Intensity

The failure intensity approach is used to study the program failure rate per fault at the failure intervals. 

As the number of remaining faults in a program change, the failure rate of the program varies accordingly.

Curve Fitting

The curve fitting is mainly applied for statistical regression analysis to study the relationship between software complexity and the number of faults in a program, as well as the number of changes.

Reliability Growth

The reliability growth is applied for the assessment and prediction to improve the program reliability through the testing process.  Reliability growth also represents the reliability or failure rate of a system as a function of time or the number of test cases. Thus, a software reliability model specifies the form of a random process that describes the behavior of software failures with respect to time.

1.7 Software Reliability Growth Models

Software reliability growth models (SRGMs) can be used to estimate and predict the number of faults that may be encountered after the software is released and provide useful information whether the software is ready to be released or required further testing. All such models use system test data to predict the number of defects remaining in the software. The utility of SRGMs can be related to its stability and predictive ability. Stability means that the model parameters should not significantly change as new data is added. Predictive ability means that the number of remaining defects predicted by the model should be closer to the number of defects found in operational field.

Parameters of SRGM are related to the total number of defects contained in a set of source code. If we know this parameter and the current number of defects discovered, we can compute how many defects are remained in the source code at a particular time shown as in figure 1.2. Residual defects are equals to the difference between total defects and number of defects discovered at a given point of time during testing.

Figure1.2 Residual defects

1.7.1 Types of Software Reliability Growth Models

Software reliability growth models can be classified broadly into two classes namely concave and S-shaped model shown as in Figure1.3. The most important thing about both the models is that they have the same asymptotic behavior, i.e., the defect detection rate decreases as the number of defects detected (repaired) increases, and the total number of defects detected asymptotically approaches a finite value (Goel and Okumoto, 1979; Yamada and Ohba, 1983).

Test Time Test Time

Figure1.3 Concave and S-Shaped Models

Thus, software reliability assessment is important to evaluate the current and predict future reliability of a software system. The models applicable to the assessment of software reliability are called software reliability growth models. Software reliability growth models (SRGMs) provide a description of software failure occurrence phenomena which can be either continuous type or discrete type SRGM.

1.7.2 Classification of Software Reliability Growth Models

In this section, we classify the software reliability growth models according to their mathematical structure, type of assumptions, method of parameter estimation, developing environment, software tools and a particular technology applied (Wood, 1996; Musa and Okumoto, 1984; Ohba, 1984) discussed as follows:

Time domain: Wall clock (months/days/hours/min./seconds) versus execution time (CPU time)

Category: The total number of failures that can be experienced in given time. This can be either finite or infinite.

Type: The distribution of the number of the failures experienced by time t. Two important types are the Poisson and Binomial.

Class: (Finite failure category only) Functional form of the failure intensity expressed in terms of time.

Family: (Infinite failure category only) Functional form of the fail­ure intensity function expressed in terms of the expected number of failures experienced.

1.7.3 Model Evaluation Criteria

The criteria for predicting software reliability for assessing and evaluating the absolute worth of a model can be described in terms of predictive validity, capability, quality of assumptions and applicability discussed as follows (Musa and Okumoto, 1983):

Predictive validity

The predictive validity is defined as the capability of the model to predict future failure behavior during either the test or the operational phases based on past failure data/behavior.

Capability

The capability refers to the ability of prediction model to assess the present status of software with satisfactory accuracy quantitatively required by the software managers and potential users. The capability of software reliability prediction model helps in designing and early development phases to plan valuable resources.

Quality of assumptions

The quality of assumptions needs to be tested quantitatively as far as possible. That is, the degree to which it is supported by actual data in order to define the clarity and explicitness of an assumption realistically.

Applicability

The applicability of a software reliability prediction model which can be judged based on its applicability across different software products, different organizations, various development environments, different operational environments, and different stages of software development.

Simplicity

The simplicity is another important characteristic of software reliability prediction model. A prediction model should be simple from the user’s point of view such as inexpensive to collect the failure data required to customize the model. The model should be able to provide sufficient guidelines to the practitioners without extensive mathematical background so that they can determine when it is applicable and the extent to which the model may diverge from reality in an application.

1.8 Software Reliability Prediction Models

Software reliability models express the probability of failure over a certain time-period for the system and time is the most commonly used metric for predicting software reliability in practice. Broadly, software reliability prediction models can be divided into two main categories namely deterministic (or static models) and probabilistic (or dynamic models).

1.8.1 Deterministic Models

A deterministic or static model uses some attributes from project or program module other than failure data to estimate the number of failures in the software. A static model of software quality estimation can be represented as: (1.2)

where, the dependent variable y is the defect rate or the number of defects. The independent variable xi is the attributes of the product, project or the process through which the product has been developed and e represents the error rate. These attributes include lines of source code, complexity, skill level, count of decision, number of modules or number of classes, weighted method per class, coupling between objects, depth of inheritance tree, and other meaningful measurements.

1.8.2 Probabilistic Models

The probabilistic reliability models gather data from the product of interest. The probabilistic reliability model tracks the failure data produced by the software system over a specified time. Probabilistic models are based on statistical distributions known as dynamic models and expressed as a function of time. All such models are applied for estimation of current and prediction of future reliability trends. Probabilistic software reliability prediction models apply statistical methods to estimate measures like the number of errors in a program, failure rate, relationship between software complexity and the number of faults in a program, and so on.

1.9 Measurement of Software Reliability

Without any agreed measure, any program of improvement is not worthy. Therefore to know what has been achieved, we must be able to test the finished product. Moreover, we cannot control what we cannot measure. Metrics are the most commonly used measurements techniques in software engineering. Two well-known and widely used metrics are categorized according to the number of failures in a time period and time between failures. Fault and failure metrics describe the product reliability characteristics like the time between failures (TBF), mean time between failures (MTBF), and failure density (FD).

However, the key problem about software metrics and measurement is that they are not consistently defined and interpreted due to the lack of physical attributes of software. Thus, reliability achieved through metrics may differ for different applications, generating non-consistent and inconclusive results. Thus, we need to identify, describe and incorporate some robust reliability-related software metrics in order to improve the quality of software systems in a realistic operating context.

We address and elaborate three main issues for the measurement of software reliability from both the customer and developers point of view as follows:

Probability: particularly with embedded software, what is the probability of mission success or how likely is an accident due to the software failure? With commercial software this is important for user-perceived reliability that how likely is the user’s work to be affected by failure?

Cost: what will the vendor’s maintenance workload be if the product is sold in its present state? This is the basis for all commercial decisions regarding maintenance contracts, warranty, size of the support teams and other resources required for maintaining software.

Fitness for release: on the basis of probability and software cost, can we release the product? If not, how much further testing is required keeping in view of time and budget constraints.

1.9.1 Software Metrics

Software metrics can be defined as "The continuous application of measurement based techniques to the software development process and its products to supply meaningful and timely management information, together with the use of those techniques to improve that process and its products" (Basili, 1985). Thus, software metrics are all about measurements involving numbers to make things better, to improve the process of developing software and to improve all aspects of the management of that process (Goodman, 1993).

1.9.2 Applications

The basic purpose of tracking fault and failure metrics is to determine when the software is attaining failure-free execution. The number of faults observed during testing phase and the failures reported by users after delivery are collected, summarized and analyzed in order to achieve better reliable software (Basili, 1981; Musa, 1987; John Harauz, 1999; Chen et al. 2001; Goel and Singh 2009).

Furthermore, software reliability metrics can determine if a product meets its reliability objectives and ready for release. Various software metrics like failure intensity as a function of test time, the number of failures expected up to testing time and mean time to failures can provide useful information in order to improve the software quality quantitatively such as:

The reliability of software systems at the end of system testing

The amount of additional test time (if required) in order to meet the reliability objectives

The reliability growth as a result of testing

The product’s reliability in the operational field if the system testing has already been done

1.10 Machine Learning Techniques

Machine learning is an approach concerned with the design and development of algorithms that allow computers to evolve the system behavior based on the statistical data of software systems. The machine learning techniques are focused on learning automatically, recognizing complex patterns and making intelligent decisions based on past data and other characteristics (Kohavi 1995; Alpaydin, 2010). So that a machine is able to learn whenever it changes its structure, program, or data based on its inputs or in response to the external information in such a manner that it’s expected future performance improves significantly.

Thus it becomes quite natural for software practitioners and researchers to know which particular potential technique tends to work well for a given failure dataset and up to what extent quantitatively.

1.10.1 Machine Learning Types

Machine learning aims to provide the knowledge engineering process, replacing much consuming time human activities with automatic techniques that improve accuracy and efficiency through discovering and exploiting regularities in failure datasets. Broadly, there are two types of machine learning techniques namely: deductive and inductive. A deductive machine learning method works on existing facts and knowledge and deduces new knowledge from the old. Inductive machine learning methods create computer programs by extracting rules and patterns from massive datasets (Breiman, 1996). Inductive learning technique learns from the examples and generalizes well with existing knowledge.

1.10.2 Applications

Machine learning techniques have wide spectrum of applications like natural language processing, search engines, medical diagnosis, classifying DNA sequences, speech and handwriting recognition, object recognition in computer vision which further can be applied to the assessment and prediction of software reliability.

1.11 Research Objectives

The objectives of this research are described as follows:

The purpose of this work is to develop the techniques and models for improving software reliability. This will lead to efficient design and development of quality software systems which can be delivered on time, within budget to the customers.

To study various significant parameters like the number of remaining faults in the software during the testing of software development, failure rate, mean time to failure, cumulative failures experienced up to a given time and other useful metrics for software reliability prediction.

To study and analyze various key issues with respect to the efficiency of metrics for software reliability growth like failure data collection, methods of parameter estimation, assumptions made during the design and development of prediction models, availability of failure data observed, in order to improve the quality of software systems.

To review and test the validity of quality metrics like failure intensity, MTTF, defect density, reliability growth factor for reliability prediction with critical inference.

To measure the comparative performance of some potential statistical and machine learning techniques which can be applied to assess the software reliability accurately?

To review and evaluate the quality metrics for software release instant using software reliability growth model.

1.12 Organization of Thesis

The rest of the work has been organized into eight chapters. The brief outline of each chapter is described as follows:

Chapter two focuses on the essentials for software reliability modeling which has been discussed thoroughly in order to meet the industry and customer needs. Software reliability prediction models play an important role in developing quality software systems. This chapter introduces some fundamental definitions, failure datasets, important terms and notations which can help software developers in preventing failures and improve the system performance resulting in better quality software.

Chapter three aims to provide the details of literature survey of existing tools, techniques and the best practices applied to the assessment and prediction of software reliability suggested by various researchers, particularly those evolved in last two decades. The significance of some well-known software metrics has been discussed in detail that can be used for the prediction of software reliability in order to enhance overall quality of software systems. In practice, the project managers and industry professionals are required to use effective prediction techniques and models to improve the quality of software systems. This chapter highlights various software reliability prediction techniques in detail focusing on methods, metrics, the current trends and existing issues. The purpose of this chapter is to help researchers and practitioners for relevant expert estimation and prediction of software reliability realistically. The findings of this chapter provide the guidance for future research to assess the impact of past and present failure data for predicting software reliability in a realistic operating environment.

In chapter four research methodology used in this thesis is described in detail. This chapter provides a detail description of the data analysis techniques, linear regression and machine learning techniques applied in this work. The validation criteria and performance measures used to evaluate the performance of predicted models are discussed in details.

Chapter five introduces a new software reliability growth models to assess and predict reliability of software systems. It is often very difficult for the project managers and practitioners to decide which model is more useful in a particular domain and up to what extent quantitatively. Moreover, the present scenario of software development has emerged into distributed environment due to the development of network technologies and ever increased demand of sharing the resources to optimize the cost. Thus, in order to improve the accuracy of reliability estimation and prediction of multi-user software systems, we consider a client server environment using three-tier architecture. This chapter proposes a NHPP based software reliability growth model (SRGM) for three-tier client server system in order to identify and remove the remaining faults in the software during the testing of software development. The proposed model is validated using seven standard failure datasets collected from various real-life and commercial projects in public domain. The software release policy has been formulated in order to stop testing and release the software to the customers by developing a software cost model. This would help software project managers and developers to ensure on-time delivery of the software meeting the criteria of attaining predefined level of reliability and development cost. A numerical example has been cited to illustrate the experimental results showing significant improvements.

Chapter six investigates several non-homogeneous Poisson process (NHPP) based models, which has been one of the most attractive reliability growth models in monitoring and tracking reliability improvement. But, the validity and relevance of assumptions to real testing environment have limited their usefulness. On other hand, learning and generalization capabilities of artificial neural networks (ANNs) and its proven success in many complex problem solutions have made it a viable alternative solution for modeling complex failure phenomena. The main purpose of using ANNs over NHPP based models is to exploit the learning and mapping capability of connectionist approach using ANN. Moreover, ANN requires only statistical failure data as input and less assumptions to make reliability predictions realistically. This chapter explores the usefulness and application of ANNs for predicting software reliability quantitatively. A comparative performance analysis is performed using feed forward neural network (FFNN), multilayer perceptron with back propagation neural network (BPNN), radial basis function (RBF), Elman neural network model in order to improve the quality of software systems.

Chapter seven implements linear regression (LR) and eight potential machine learning techniques (ANNs, SVMs, FIS, ANFIS, GEP, GMDH, MARS, and DTs) for predicting software reliability based on past failure behavior of software systems. The machine learning techniques are focused on learning automatically, recognizing complex patterns and making intelligent decisions based on past experience and other statistical observations. So that a machine is able to learn whenever it changes its structure, program, or data, based on its inputs or in response to the external information in such a manner that it’s expected future performance improves. The applications of machine learning techniques for the prediction of software reliability in place of statistical techniques have shown remarkable improvement in recent years. Therefore it is desirable and interesting to analyze and compare between LR and machine learning techniques in practice. Accurate software reliability prediction can not only enable developers to improve the quality of software but also provide useful information to help them for planning valuable resources. The effectiveness of LR and machine learning techniques are illustrated with the help of sixteen failure datasets taken from the Data and Analysis Centre for Software (DACS). A comparative analysis is performed in order to assess the predictability of the proposed models in a realistic operating environment.

Chapter eight summarizes the findings of our work with some possible future directions and limitations. This chapter concludes the salient findings of this thesis and briefly recapitulate the outcomes of our work.

CHAPTER 2

ESSENTIALS FOR SOFTWARE RELIABILITY PREDICTIONS

Introduction

Software reliability prediction models play an important role in developing software systems and enhance the quality of software systems. The software reliability prediction techniques and models used in this work deals with the probabilistic methods applied to the analysis of random occurrence of failures in given software. This chapter introduces some fundamental definitions, important terms and notations, commonly used failure datasets and widely used software reliability prediction tools in practice.

2.2 Related Definitions

There are large number of terms and definitions used in software reliability engineering. This section presents some specific statistical measures considered useful in classical and modern reliability modeling discussed as follows:

Reliability, R (t) is the probability that a system will be successful in the interval from time 0 to time t which may be defined as:

(2.1)

where T is a random variable denoting the time-to-failure or failure time (Musa, 1987; Aggarwal, 1993; Trivedi K., 2001; Ebeling, 2007). Alternatively, the failure rate F(t), can be defined as the probability that the system will fail by time t can be written as:

(2.2)

In other words, F (t) is the failure distribution function and the relationship between reliability and failure rate can be defined as follows:

(2.3)

2.2.1 Probability Definition

Probability may be defined as likelihood of occurrence of a given event (Trivedi K., 2001). Mathematically, probability can be represented as:

(2.4)

where P is the probability.

Random Variable

Random variable (RV) is a variable quantity which denotes the result of a given experiment known as stochastic variable. RV is denoted by upper case letters and their realizations by the corresponding lower case. A random variable may be either discrete or continuous. A random variable X is known as discrete if its range forms a discrete (countable) set of real numbers. On the other hand, random variable X is known as continuous if its range forms a continuous set of real numbers and the probability of X equalling any single value in its range is zero (Pham, 2006; Aggarwal and Singh, 2008).

Here we consider a variable X that can take on only the values with the respective probabilities then the variable X over its complete range of values can be described as: (2.5)

where x is called a random variable. The function P(x) which takes on the discrete values at the given points is called the probability frequency or probability density function given by: (2.6)

The probability distribution function can be described as follows:

(2.7)

where X refers to the set of all , is the probability frequency function and is probability distribution function (Trivedi, 2001).

The reliability quantities are usually defined with respect to time. Here we describe different types of time variables such as execution time, calendar time and clock time variables (Musa, 1987; 2007; Pham, 2006) used in our study as follows:

The execution time for a program is the time that is actually spent by a processor in executing the instructions of that program.

The calendar time is the familiar time that we normally used in practice and can be represented in terms of hours/days/weeks/months or years.

The clock time is used for a program which represents the elapsed time from start to the end of program execution on a running computer. It includes wait time and the execution time of other programs.

Faults and Failures

The failure occurrences are expressed using random variables. Since the values of such variables are not known with certainty. There could be many possible values each associated with a probability of occurrences. Therefore, we do not really know when the next failure will occur. We only know a set of possible times of failure and its probability of occurrence.

The reason for this randomness is due to the commission of errors and introduction of faults which is a complex and unpredictable process. Hence, the locations of faults within the program are unknown. Some basic terminology used in this thesis for the assessment and prediction of software reliability is described as follows:

Errors: are human actions that result in the software containing a fault. The error is the cause of a failure, which is unacceptable departure from nominal program operation. Examples of such faults include the omission or misinterpretation of the user’s requirements, or coding error etc. (Musa, 1987; Lyu, 2005; Aggarwal and Singh, 2008).

Software error: an error committed by a programmer or designer like typographical error or an incorrect numerical value or an omission, etc.

Faults: are the incorrect logic/instruction, or inadequate instruction that upon execution will cause a failure. Thus, faults are the manifestations of an error in the software. If encountered then it may cause a failure of the software. Bugs are synonymous with fault.

Failure: is the inability of the software to perform its functions within specified limits. Failures are observed during the testing and operational stage of software development.

Faults …………Errors Failures

Failure rate: refers to the hazard rate or rate of occurrence of failure (ROCOF) depending upon the context. The hazard rate is the conditional instantaneous rate of failure at a point in time, given that the software has operated without failing up to that time. The ROCOF is the unconditional rate of occurrence of a failure at a point in time.

Software failure: a failure caused by a software fault. Moreover, software itself does not fail. Faults are already present in the software that leads to the failure of system under certain conditions.

Operational profile: a set of operations that software can execute, given the probabilities of their occurrence.

2.3 Measures of Reliability

Some important key functions to describe the failure process of a software system are discussed as below (Ebeling, 2007). A failure process, represented by the random variable T (time to failure) can be described in terms of: (i) the probability density function (ii) the cumulative distribution function (iii) the reliability function or (iv) the failure intensity function

2.3.1 Probability Density Function (PDF)

The probability density function is defined by, where t is the time to failure (using continuous random variable). The probability density function is often referred to as failure density function.

2.3.2 Cumulative Distribution Function (CDF)

The probability density function f(t), the cumulative distribution function F(t), the reliability function R(t) and the failure intensity function λ(t) can be related using the following equations.

and

(2.8)

A random variable may be defined over the open interval (-∞, ∞), but when the random variable represents a failure time, only nonnegative values are allowed. Therefore the domain of the random variable is taken as [0, ∞). Under general conditions, any one of these functions can be determined from the given λ(t), and R(t).

If the intensity function increases, then the probability of failure over a specific interval of time becomes greater as the testing time proceeds. This trend indicates that software system deteriorates. On the contrary, if the intensity function decreases, this indicates that the software reliability is growing. Although, in software systems it is reasonable to assume that the intensity function may change only when the program undergoes some modification in its source codes (addition of new source codes in the program or fault removal, and so on).

2.3.3 Mean Value Function

The mean value function represents the average cumulative failures at each point in time. Software failures can be characterized using failure occurrence time or number of failures occurring at specific time. Here, we consider m(t) a random process which represents the failure occurrence at time t. The expected number of failures at time t is computed by using mean value function as below: (2.9)

2.3.4 Failure Intensity Function

The failure intensity function, λ(t) is the rate of change of the mean value function or the number of failures per unit time (Goel and Okumoto, 1979). Failure intensity function of m(t), quantifies the rate of change of expected number of faults at time t computed as follows:

(2.10)

Thus, failure intensity is the first derivative of the mean value function with respect to time and known as an instantaneous value. A random process whose probability distribution varies with time is called non-homogeneous process. The failure intensity is an alternative way of expressing reliability and it is more economical also, as we only have to provide the number(s) to compute reliability (Musa, 2007; Aggarwal and Singh, 2008).

2.4 Empirical Failure Data Sets

The main purpose of collecting software failure data is to make sure that collected data can provide useful information for researchers’ in order to improve the quality of software systems. In software reliability modeling mainly two types of failure data namely time-domain data and interval-domain data has been widely used. The category of time-domain data is characterized by recording the individual times, when the failure occurred. The interval-domain data are characterized by counting the number of failures occurring during a fixed period. Some of the specific and widely used failure datasets of public domain available in literature are described in Table2.1

Table2.1 Empirical failure datasets

Datasets

Description

No. of failures/defects

Sources

DS1

On-line IBM entry software package: A small on-line data entry software package test data, available since 1980 in Japan (Ohba 1984a). The size of the software is approximately 40,000 LOC. The testing time was measured in terms of the number of shifts spent running test cases and analyzing the results. This category of failure dataset has been used by many researchers for testing and validating software reliability prediction models.

46

Ohba (1984)

DS2

On-line Communication System (OCS): The On-line Communication System (OCS) project at ABC Software Company was completed in 2000 (Pham 2003a). The project consisted of one unit-manager, one user interface software engineer, and ten software engineers/testers. The overall effort was divided into four phases in the software development process of the project such as analysis, design, coding and testing respectively.

55

Pham (2003)

DS3

This set of failure data is taken from Misra (1983), consists of three types of errors: critical, major, and minor. The observation time (week) and the number of failures were recorded weekly.

86

Misra (1983)

DS4

US Naval Tactical Data Systems (NTDS): This failure dataset is extracted from the development of software for the real-time multi-computer complex of the US Naval Fleet Computer Programming Center of the US Naval Tactical Data Systems (Goel 1979a). The software consists of 38 different project modules. The time horizon is divided into four phases: production phase, test phase, user phase, and subsequent test phase. The 26 software failures were found during the production phase, five during the test phase and; the last failure was found on 4 January 1971. One failure was observed during the user phase, in September 1971, and two failures during the test phase in 1971.

34

Goel (1979)

DS5

Tandem Computers Software Data Project: This failure dataset from release #1 has been taken from one of the four major releases of software products at Tandem Computers (Wood 1996).

100

Wood (1996)

DS6

On-Line Data Entry IBM Software Package: This failure dataset was reported by Ohba (1984a) and recorded from testing the on-line data entry software package developed at IBM.

15

Ohba (1984a)

DS7

AT&T System Project: The AT&T's System is a network-management system developed by AT&T that receives data from telemetry events, such as alarms, facility-performance information, and diagnostic messages, and forwards them to operators for further action. This failure dataset was collected by Ehrlich in 1993.

22

Ehrlich (1993)

DS8

Real-Time Control Systems (Hou et al., 1997): The software for monitor and real-time control systems (Tohma 1991) consists of about 200 modules and each module has, on average, 1000 lines of a high-level language like FORTRAN. The software failure records were detected during the 111-day testing period. This failure dataset was recorded with several up and downs reflecting different clusters of detected faults. This datasets have been used by several researchers and many applications.

481

Hou et. al. (1997)

DS9

The Real-time Control System Data: This failure dataset was documented in Lyu (1996). There are in total 136 faults reported and the time-between failures (TBF) in second during testing. This dataset has been utilized by many potential users for design and validation of new models.

136

Lyu (1996)

DS10

Real-Time Command and Control System: This failure dataset was reported by Musa (1987) based on failure data from a real-time command and control system, which represents the failures observed during system testing for 25 hours of CPU time. The delivered umber of object instructions for this system was 21700 and was developed by Bell Laboratories.

136

Musa (1987)

DS11

This failure data set was reported by Zhang (2002) based on system test data for a telecommunication system. System test data was reported in two releases (Phase 1 and 2). In the tests, automated test and human- involved tests were executed on multiple test beds.

43

Zhang

(2002)

2.5 Tools for Software Reliability Modeling

Over the years a significant number of software reliability tools have been developed for reliability prediction in the literature. These tools provide a general framework for reliability estimation and prediction (Musa, 1983; 1989). We categorize the reliability prediction tools based on their functionality, capability, availability, applicability and user interface.

Further, software reliability tools can also be characterized using other parameters like collection of failure data, estimation of model parameters, testing the fit of a model against the collected information and application of the model (Littlewood, 1989; Wood, 1996). Some well-known and widely used software reliability prediction tools in practice are summarized in Table2.2.

Table2.2 Software reliability modeling tools

S. No.

Software reliability Tools

Description and Application

1

AT&T

AT&T Software Reliability Engineering Toolkit

The AT&T Software Reliability Engineering Toolkit is a command line driven system which executes Musa basic and Musa-Okumoto software reliability models. This tool can be applied for both time-domain and interval-domain failure datasets for predicting software reliability.

2

SMERFS

Statistical Modeling and Estimation of Reliability Functions for Software

This tool is basically applied for Statistical Modeling and Estimation of Reliability Functions for Systems. SMERFS was designed by William Farr (1982) and it has been utilized in many areas of software reliability engineering ever since. SMERFS is a program for estimating and predicting the reliability of software during the testing phase. It uses failure count information to make reliability predictions. SMERFS allows the user to perform risk analysis and to determine the optimum release time of the software.

3

SRMP

Statistical Reliability and Modeling Programs

The Statistical Reliability and Modeling Program was developed by reliability and statistical consultant in 1988. SRMP is a command-line interface tool which contains nine models for predicting software reliability from failure datasets. The parameters are estimated using Maximum Likelihood Estimation (MLE) method in order to provide the reliability information such as failure rate, mean time to failure, median time to failure etc. SRMP accepts only time-domain input data for making predictions.

4

ESTM

Economic Stop Testing Model

Economic Stop Testing Model is a command-line interface tool which can be applied for releasing the software. It determines the optimal stopping time using a birth-death model for the introduction of faults and a cost-benefit model to measure the trade-off between whether software can be released or further testing is required. Moreover, it helps the user when to stop the testing and release the product for operational use based on reliability achieved.

5

SoRel

Software Reliability Program

Software Reliability Program is a measurement tool developed by Lab of The National Center For Scientific Research, France in 1991. The SoRel has the capability of testing and analyzing the data for making predictions with better accuracy. Several types of tests can be conducted using SoRel leading to more accurate reliability estimation and predictions. It allows various models to accept both time-domain and interval-domain input data which can be applied for predicting software reliability growth.

6

SARA

Software Assurance Reliability Automation

Software Assurance Reliability Automation (SARA), also known as Non-Parametric Software Reliability tool is a comprehensive system for predicting software reliability. SARA incorporates both reliability growth model metrics and design code metrics for analyzing the software applicable to time between failure data. Initially, SARA was a research project funded by the NASA and supported by Software Assurance Technology Center (SATC).

7

CASRE

Computer-Aided Software Reliability Estimation

Computer-Aided Software Reliability Estimation Tool is well-known and widely used software which was developed by Jet Propulsion Laboratories in 1993. CASRE is a user friendly tool with GUI features which runs on Windows operating system. CASRE is capable to customize the user’s model using in-build models and can be applied to time-domain failure datasets.

8

RGA

ReliaSoft

RGA: Reliability Growth Analysis and Repairable System Analysis Software Tool

RGA is product of ReliaSoft's, particularly applied to software reliability growth models for analyzing data from both developmental testing and fielded repairable systems. In the development stage, RGA can be utilized to quantify the reliability growth achieved with each successive design prototype and also provides advanced methods for reliability growth projections, planning and management. RGA also supports fielded repairable system analysis, including a Design of Reliability Test (DRT) utility and a method for analyzing the system's reliability behavior in order to calculate the optimum overhaul times and other useful metrics of interest.

9

Weka

WaiKato Machine Learning Tool

The Waikato Environment for Knowledge Analysis (Weka), a comprehensive tool composed of Java class libraries has been applied for implementing several machine learning algorithms. Weka is freely available open source suite which can be accessed from http://www.cs.waikato.ac.nz of "Machine Learning Group", University of Waikato.

10

DTReg

Classification and Regression Tool

DTREG is predictive modeling software developed by Phillip H. Sherrod (2003). DTREG has a huge capability of modeling various classification and regression decision trees using neural networks, support vector machine (SVM), GMDH polynomial networks, gene expression programs, K-Means clustering, discriminant analysis and logistic regression models that describe data relationships and can be used to predict values for future observations. DTREG can also be applied for time series analysis. The details of DTReg can be accessed through www.dtreg.com

11

Salford System SPM

Software Predictive Model suite

Salford System: SPM is a software predictive modeling site with a proven record of technical and practical excellence. SPM suite offers a highly accurate predictive analytics and data mining platform for developing models from databases of variable size, complexity, or organization. SPM suite support several data mining techniques such as classification, regression, missing value analysis, and clustering/segmentation. SPM suite contains four basic modules including CART, MARS, TreeNet and Random Forests. Further details can be found at http://www.salford-systems.com

12

SVM Tools

Support Vector Machine Tools

The university of Wisconsin, Madison provides several effective support vector machines (SVM) classification software that has been developed by researchers at the Data Mining Institute as well as a data generator. A list of support vector machine tools such as SSVM, PSVM, LSVM, NDC and other useful tools can be found at http://www.research.cs.wisc.edu/dmi/svm

2.6 Discussion

The project managers and industry professionals are required to use effective prediction techniques in order to improve the quality of software systems. This chapter highlighted several essential terms, definitions in detail focusing on modeling complex failure phenomena, failure datasets and some commonly used and commercially available software reliability estimation tools in practice. The purpose of this chapter is to help researchers and practitioners for relevant expert estimation and prediction of software reliability realistically. The findings of this chapter provide the guidance for future research to assess the impact of past and present failure data for predicting software reliability in a realistic operating context.

CHAPTER 3

LITERATURE SURVEY*

3.1 Introduction

The primary objective of reliability prediction is to assess future reliability of software systems from its design or specification phase (Musa, 1975). Therefore, researchers are more focused on reliability assessment to avoid any unforeseen situation at later stage during operational phase. As our global society becoming more dependent on information in the production of goods and services, the pressure for high quality, low cost, and fast delivery of software systems are increasing. However, the quality of software, cost of the software development, and schedules are conflicting characteristics. The customers can't get one without paying for another.

This chapter aims to provide the detailed literature survey of existing techniques and best practices applied for the assessment and predictions of software reliability suggested by various researchers and practitioners particularly those evolved in last two decades. The pros and cons of some potential techniques for predicting software reliability has been discussed in detail and reviewed thoroughly. The significance of some potential software metrics has been discussed in detail, which can be used for the assessment and prediction of software reliability in order to enhance the overall quality of software systems.

Significant Developments

Over the years many software reliability growth models have been employed for predicting reliability. There is no universal agreement among the researchers and practitioners in the field of software reliability engineering that a correct or best model can exist. Because one modeller might consider certain aspects of reliability very important thus giving them significant weightage in his or

*The major findings of this chapter have been accepted for publication in the International Journal of Information and Communication Technology (IJICT), Inder-Science Publication, January 2013.

her model. On the other hand, another modeller may have dissimilar views which result in a different model.

Jelinski and Moranda (1972) proposed the first Software Reliability Growth Model (SRGM) and since then a significant number of SRGM have been proposed in the past forty years like exponential failure time class models, Weibull and Gamma failure time class models, infinite failure category models, Bayesian models, and so on. However, the major challenges of these models do not lie in their technical soundness, but their validity and applicability in real world scenario. Two well known and widely used SRGM applied for the assessment and prediction of reliability are G-O model discussed as below.

The Goel-Okumoto Model (1979), is a category of concave SRGM which can be represented as:

, for and (3.1)

where, μ (t) = expected number of defects at time t

t = execution time, or number of tests executed, which can be represented in terms of calendar time

a = expected total number of defects in the source code

b = shape factor or the rate at which the failure rate decreases

Another category of SRGM is the modified Goel-Okumoto (G-O) model which is known as S-shap