The Common Vulnerabilities Scoring System

Published Date: 02 Nov 2017

Computer and network security or information security is considered as a critical issue for organizations at all times. Numerous flaws and weaknesses, known as vulnerabilities,exist in the software products that can be exploited to compromise the organization security. The Common Vulnerability Scoring System (CVSS) is the most widely used and publically available scoring system for measuring the severity of the security vulnerabilities. US National Vulnerability Database (NVD) uses CVSS to quantify the severity of these security vulnerabilities. But CVSS has not been validated so far with empirical results. Many different models have been presented for improving the CVSS but all of them provide only qualitative discussion for such improvements. This research conducts an experts’ survey to obtain their suggestions and judgment about the validity and reliability of the CVSS.

Keywords:Common Vulnerability Scoring Metrics (CVSS), Computer and IT Security, Security vulnerabilities.

Table of Contents

INTRODUCTION

BACKGROUND

The computer has become an essential part of the modern society all around the world and they cannot think a single day without a computer. Every day life depends on the computer system. Shopping malls, restaurants, traveling by air or by train and banks etc., Everywhere a person will come across a computer system in one or other form. Therefore the security of such computer systems and security of information within these systems is considered as a critical issue for organizations at all times. Information and/or data in these systems is a lifeline for organizations and unauthorized access or destruction of such information or data can bring disaster to organizations and can make them incapable of carrying out their normal functions.

That is why Information and information technology is always endanger and vulnerable to security risk. It is the most important activity for all levels of the organization to measure and manage such security vulnerabilities. Measuring and managing these security risks is the most challenging tasks that an organization faces. Therefore how to measure and how to manage these security vulnerabilities, is always a top priority of security professionals in organizations [3].

In computer security, the vulnerability is defined as the flaw or the weakness in the system that could be exploited to violate the system's security policy [10]. A large amount of resources is utilized in order to overcome and/or reduce security threats due to these vulnerabilities [3, 9]. Exploitation of these security vulnerabilitiescan be very fatal in term of availability, confidentiality and integrity. Organizations are always in search of methods through which security breaches could totally or partially be avoided.

Software flaws (vulnerabilities) are numerous, and thousands of new ones are discovered every year in commonly used software products [3, 9, 11]. A number of approaches have been presented for measuring the severity of the security vulnerabilities. Such as [1, 2, 3, 4, 5, 6, 12, 13, 14, 15, 16]. Security measuring metrics are accepted widely and are considered very important in the information security, because it is a difficult task for the security professional to quantify the success of implemented security mechanism without these security measuring metrics [9]. Therefore it is obvious that if a security risk cannot be measured then, in turn, it cannot be improved. That is why security professionals consider these measuring metrics very effective tools for implementing a successful security plan and mechanism.

The Common Vulnerability Scoring System is the de facto standard for comparing the severity of IT security vulnerabilities[8]. CVSS provides metrics and describes an algorithm for measuring the severity of security vulnerabilities. National Vulnerability Database, NVD, is the U.S. government repository of standards based vulnerability management data and contains more than 50000 vulnerabilities (September 2012). NVD uses CVSS for measuring the severity of security vulnerabilities [7]. Keeping these vulnerabilities in mind, security management is a crucial activity in organizations’ security efforts. It is very important for organization to rank these security vulnerabilities according to their severities in order to prioritize the security protection plan and mechanism.

RESEARCH PROBLEM

As mentioned in section 1.1 that, every year, numerous security vulnerabilities are discovered in software products. Organization and enterprises utilize their resources to find and measure the severity security vulnerabilities. Different measuring algorithms are used to measure the severity of these security vulnerabilities in order to implement a useful security mechanism. Common Vulnerability Scoring System is one of the severity measuring systems for measuring security vulnerabilities. CVSS provides information based on various attributes and an algorithm to quantify the severity of security vulnerabilities. CVSS is most commonly used scoring system and many organizations are using this system to measure the severity of the software vulnerabilities [8]. By measuring the severity of vulnerabilities, organization prioritizes the vulnerabilities within the software so that they can deal with the most severe and fatal ones first. National Vulnerability Database (NVD), U.S government repository of security vulnerabilities, contains more than 50000 vulnerabilities and it uses CVSS algorithms to measure the severity of these vulnerabilities. Similarly many organizations have also adopted CVSS to prioritize their protection mechanism and security plan. The central point of the above discussion is that; is there any research present that could evaluate the validity and reliability of the information and the algorithm provided by CVSS. Because it is very important to evaluate the information and the algorithm of CVSS as it is the most widely used scoring system all over the world [8] and therefore any flaw in CVSS system could lead to weak security plan and weak protection mechanism. If such security plan and mechanism are breached, it can be very fatal for the organization and can bring disaster to organizations.

Security vulnerabilities are the flaws or weakness in the software that can be exploited to violate the system's security policy. Thousands of new security vulnerabilities are found every year in common software. These security vulnerabilities need to be measured in order to prioritize the security mechanism. A huge amount of recourses is used to find and measure the security vulnerabilities in software. Since the exploitation of these security vulnerabilities can be very fatal in term of availability, confidentiality and integrity. Organizations are always in search of methods through which they can measure the severity of vulnerabilities so that security breaches could totally or partially be avoided. Since the NVD is a U.S government database of vulnerabilities and it contains more than 50000 security vulnerabilities. And CVSS is publicly available scoring system for measuring the severity of vulnerabilities. Therefore NVD and many other organizations use CVSS for measuring the severity of security vulnerabilities. But CVSS has not been validated so far with empirical results. Many researchers have proposed different models for improving the CVSS but all of them provide only theoretical conclusion for such improvements. No quantitative data are available for these models to make them valid and reliable. Therefore lack of such quantitative data (empirical study) suggests that quantitative approaches, other than observation, are needed. Common Vulnerability Scoring System CVSS is the most widely used system that provides information and an algorithm for measuring the severity of security vulnerabilities found in software systems.

RESEARCH QUESTION

This thesis research studies the CVSS to evaluate the validity and reliability of its current scoring system. The name of the Master’s thesis is "An evaluation of the Common Vulnerability Scoring System. This research presents a method for evaluating the current CVSS scoring system..

This thesis research studies the following research questions:

Is the current scoring system of CVSS reliable? Meaning that whether the algorithm of CVSS is sufficiently reliable for measuring the severity of the security vulnerabilities?

Is the information for measuring the severity of vulnerabilities provided by CVSS valid? The question can be further broken down as

Are the attributes in the current CVSS system sufficient for estimating the severity of vulnerabilities

Are there any attributes that need to be added or removed?

After the analysis of the data collected from experts, this thesis will present a new model of CVSS for measuring the severity of the security vulnerabilities in a more reliable way.

METHODOLOGY

Methodology in research provides a guideline system for solving a problem and is an important part of the research. This guideline system consists of different components such as techniques, methods, phases and tools for solving research problems. The methodology this research uses is discussed in chapter 2 in detail which will be used for solving the research questions mention in section 1.2. Following is a brief overview of the methodology that will be used.

As mentioned earlier in this chapter that expert (expert sampling is explained in chapter 2) judgment is needed for the validity and reliability of the current CVSS Scoring System. To obtain expert judgment and their suggestion, this thesis research conducts a survey with a questionnaire. It is very important to keep in mind that the participants (the respondents) should be security experts and testers in order to get a very reliable response. Sampling the security personnel for this thesis research is explained in chapter 2. It is also very important to measure their (respondents) expertise as experts may have different responses for different vulnerabilities. So as to make a consensus among the experts for a set of questions and could be a sufficient indicator of the expert performance. An online survey will be designed (PHP/MySql) with questionnaires to collect experts’ judgments and suggestions. Since National Vulnerabilities Database NVD contains more than 50000 vulnerabilities, therefore a sample will be selected from this NVD by choosing vulnerability in a random fashion. The selected vulnerability will be presented to the respondent with all information that NVD has about that particular vulnerability. But the actual identity of the vulnerability will not be shown to the respondent. In this way data will be collected for further analysis. A complete methodology description is provided in chapter 2 of this thesis report.

AUDIENCE

Sometime it is very confusing for a researcher to figure it out that who will be the potential audience for his/her research. Should the researcher consider the instructor as an audience or the whole academic community? First one is too narrow while the latter one is too broad. Therefore it is very important for the researcher to find his/her audience that lies somewhere in between these two extreme. The style and tone of the research depend on the audience of that research. It is very important to know that research adds a new knowledge to the research field and to the specific academic community of that field. This research evaluates the CVSS therefore the potential audience of this research are the security managers, experts and security policy making personals.

COMMON VULNERABILITY SCORING SYSTEM

Information and information technology is always endanger and vulnerable to security risk. It is the most important activity for all levels of the organization to measure and manage such security vulnerabilities. Measuring and managing these security risks is the most challenging tasks that an organization faces. Therefore how to measure and how to manage these security vulnerabilities, is always a top priority of security professionals in organizations [3]. As there are so many vulnerabilities to fix and there are many scoring systems for measuring the severity of the security vulnerabilities. Such as [1, 2, 3, 4, 5, 6, 12, 13, 14, 15, 16 ]. Security measuring metrics are accepted widely and are considered very important in the information security, because it is a difficult task for the security professional to quantify the success of implemented security mechanism without these security measuring metrics [9]. But how can IT and security managers convert such huge amount of vulnerability data into useful information?. The Common Vulnerability Scoring System is the de facto standard for comparing the severity of IT security vulnerabilities [8], and is an open framework that addresses this issue. CVSS provides metrics and describes algorithms for measuring the severity of security vulnerabilities. National Vulnerability Database, NVD, is the U.S. government repository of standards based vulnerability management data and contains more than 50000 vulnerabilities (September 2012). NVD uses CVSS for measuring the severity of security vulnerabilities [7]. A brief introduction of CVSS and how it quantifies the severity of any vulnerability is given below.

BACKGROUND

The Common Vulnerability Scoring System (CVSS) is a specification for documenting the major characteristics of vulnerabilities and measuring the potential impact of vulnerability exploitation [8]. The motivation for developing CVSS was to provide standardized information for organizations to use to prioritize vulnerability mitigation. CVSS is developed and maintained by the CVSS Special Interest Group (CVSS-SIG) working under the auspices of the Forum for Incident Response and Security Teams (FIRST). The U.S. Federal government uses it for its National Vulnerability Database [8] and mandates its use by products in the Security Content Automation Protocol (SCAP) validation program [77]. CVSS has also been adopted by dozens of software vendors and service providers [78]. There are many proprietary schemes for scoring software flaw vulnerabilities, most created by software vendors, but CVSS is the only known open specification. CVSS is also distinguished from other scoring systems in that CVSS was designed to be quantitative so that analysts would not have to perform qualitative evaluations of vulnerability severity [79]. In addition, CVSS is designed to provide visibility into how a score was calculated. Each CVSS score is provided with a CVSS vector. This vector includes metrics that categorize several characteristics of vulnerability. The vector provides details on the nature of the vulnerability that help CVSS users to understand why vulnerability received a particular score. These two attributes of CVSS, quantitative analysis and transparency through vectors, lend the specification to research and analysis. Large publicly available CVSS data sets from the National Vulnerability Database [8] further enable this research. The initial CVSS specification was developed by the National Infrastructure Advisory Council [80] and published in October 2004. As [81] explains, the original specification did not undergo widespread peer review, and adopters raised several concerns about it. The CVSS-SIG worked from April 2005 to June 2007 on identifying problems.

INTRODUCTION

CVSS uses three groups of metrics to calculate vulnerability scores: Base, Temporal and Environmental, each consists a set of metrics, as shown in Figure 2.1 [8], and Figure 2.2 [8] shows the CVSS framework to calculate the scores of vulnerabilities.

AccessVector(AV)

AccessComplexity

(AC)

ConfImpact (C)

IntegImpact (I)

AvailImpact(A)

Authentication (Au)

CollateralDamagePotential

Target Distribution

ConfidentialityRequirement

IntegrityRequirement

AvailabilityRequirement

Exploitability

RemediationLevel

ReportConfidence

CVSS Metrics Groups

Base Metrics Group

Environmental Metrics Group

Temporal Metrics Group

Figure 2.1 : CVSS Metrics Groups

BaseMetrics

Temporal Metrics

EnvironmentalMetrics

Optional

CVSS Score

Figure 2.2: CVSS Framework

Base metrics:

Base metrics are vulnerability attributes that are constant over time and across all implementations and environments. Base metrics group consists of sub metrics Impact and Exploitability.

Temporal metrics

Temporal metrics are vulnerability attributes that change over time but which apply to all instances of vulnerability in all environments (e.g., the public availability of exploit code or a remediation technique). A temporal score for vulnerability is calculated with an equation that uses both the base score and temporal metric values as parameters.

Environmentalmetrics

Environmental metrics are vulnerability attributes that are organization and implementation specific, such as how prevalent a target is within an organization. An environmental score is calculated with an equation that uses both the temporal score and the environmental metric values as parameters.

There are six base metrics in CVSS v2 in the Base Metrics Group as shown in Figure 2.1. The first three metrics relate to exploitability. AccessVector measures the range of exploitation (e.g., can it be launched over the network or only locally). Authentication measures the level to which an attacker must authenticate to the target before exploiting the vulnerability. AccessComplexity measures how difficult it is to exploit the vulnerability once the target is accessed. These three metrics, which collectively measure how readily an attacker can attempt to exploit vulnerability, comprise an exploitability subvector from which exploitability subscore can be calculated.

In addition to the three exploitability metrics, v2 also has three base metrics related to impact. ConfImpact measures the level to which vulnerability exploitation can impact the target’s confidentiality, and IntegImpact and AvailImpact capture the same information for integrity and availability, respectively. The impact metrics collectively measure the extent to which an attacker can compromise a computer’s security by exploiting a particular vulnerability. The three impact metrics form the impact subvector, from which an impact subscore can be determined. Table 1 lists the possible values for each metric in CVSS v2, along with the abbreviations (in parentheses) for each metric and metric value.

Table 2.1: Metrics and Values of CVSS Base Score Metric

Nameofmetric

Values

Description

AccessVector (AV)

Network (N): 1.0

Adjacent network (A): 0.646

Requireslocal access (L): 0.395

AccessVector measures the range of exploitation (e.g., can it be launched over the network or only locally).

AccessComplexity

(AC)

Low (L): 0.71

Medium (M): 0.61

High (H): 0.35

AccessComplexity measures how difficult it is to exploit the vulnerability once the target is accessed

Authentication

(Au)

Not required (N): 0.704

Single instance (S): 0.56

Multipleinstances (M): 0.45

Authentication measures the level to which an attacker must authenticate to the target before exploiting the vulnerability

ConfImpact (C)

Complete (C): 0.660

Partial (P): 0.275

None (N): 0.0

ConfImpact measures the level to which vulnerability exploitation can impact the target’s confidentiality

IntegImpact (I)

Complete (C): 0.660

Partial (P): 0.275

None (N): 0.0

IntegImpact measures the level to which vulnerability exploitation can impact the target’s integrity

AvailImpact (A)

Complete (C): 0.660

Partial (P): 0.275

None (N): 0.0

AvailImpact measures the level to which vulnerability exploitation can impact the target’s availability

HOW CVSS WORKS

Base score is assessed through Equations 1-4 and uses a scale from 0-10 where 0.0-3.9 means Low Severity Vulnerability, 4.0-6.9 means Medium Severity Vulnerability and 7.0-10 is considered as High Severity Vulnerability. The Equations 1-4 are given below. To calculate the v2 base score, the three exploitability metrics are combined into an exploitability subscore using the Equation 3. The three impact metrics are combined into impact subscore using the Equation 2. The base score is calculated from the subscores using the following Equation 1.

BaseScore = round_to_1_decimal ((0.6 * Impact + 0.4 * Exploitability – 1.5 * f (Impact))) (1)

Imppact= 10.41 * ( 1- ( 1 – ConfImpact) * (1 – IntegImpact ) * ( 1 – AvialImpact) ) (2)

Exploiatability = 20 * AcesssComplexity * Authentication * AcessVector(3)

f(Impact) = 0 if impact = 0; 1.176 otherwise (4)

When the base metrics are assigned values, the base equation calculates a score ranging from 0 to 10, and creates a vector, as illustrated below in Figure 2.2.

Optionally, the base score can be refined by assigning values to the temporal and environmental metrics as shown in Figure 2.2. This is useful in order to provide additional context for vulnerability by more accurately reflecting the risk posed by the vulnerability to a user’s environment. However, this is not required. Depending on one’s purpose, the base score may be sufficient.

If a temporal score is needed, the temporal equation will combine the temporal metrics with the base score to produce a temporal score ranging from 0 to 10. Similarly, if an environmental score is needed, the environmental equation will combine the environmental metrics with the temporal score to produce an environmental score ranging from 0 to 10.

BASE, TEMPORAL, ENVIRONMENTAL VECTORS

Each metric in the vector consists of the abbreviated metric name, followed by a ":" (colon), then the abbreviated metric value. The vector lists these metrics in a predetermined order, using the "/" (slash) character to separate the metrics. If a temporal or environmental metric is not to be used, it is given a value of "ND" (not defined). The base, temporal, and environmental vectors are shown below in Table 2.. For example, a vulnerability with base metric values of "Access Vector: Network, Access Complexity: Low, Authentication: Single instance, Confidentiality Impact: Partial, Integrity Impact: Complete, Availability Impact: None" would have the following base vector: "AV:N/AC:L/Au:S/C:P/I:C/A:N."

Table 2.2: Base, Temporal and Environmental Vectors

Metric Group

Vector

Base

AV:[L,A,N]/AC:[H,M,L]/Au:[M,S,N]/C:[N,P,C]/I:[N,P,C]/A:[N,P,C]

Temporal

E:[U,POC,F,H,ND]/RL:[OF,TF,W,U,ND]/RC:[UC,UR,C,ND]

Environmental

CDP:[N,L,LM,MH,H,ND]/TD:[N,L,M,H,ND]/CR:[L,M,H,ND]/

IR:[L,M,H,ND]/AR:[L,M,H,ND]

RELATED WORK

This chapter reviews the literature in the thesis subject area. Related work is a critical part of any research work. It provides the bases on which the research question is justified and the research design is built [19]. The related work provides information how to carry out the research and collect data. It also enables how to analyze the data in an informed way. The importance of the reviewing literature is to know what is already known about the area of interest. The related work shows the work of others in the same field as the researcher. Reviewing the literature is not only to reproduce the theories and opinion of other scholars but also being able to interpret what they have written, possibly by using their ideas to support a particular view point or argument [19]. Therefore literature review is a process of identifying relevant information by going through sources e.g. articles, databases. Literature review is a mean of showing why the research questions are important. It is important to note that reading and reviewing the literature is not something that one should stop doing once the research is begun [19]. The researcher should continue the search for and reading of the relevant literature throughout the research. A literature search relies on careful reading of books, journals and reports [19]. Once the keywords, that help to define the boundaries of the research area, are identified then the electronic databases of published literature can be searched for previously published work in the field.

This chapter reviews and describes such articles that are relevant to this thesis research work. The search approach for published articles to review is to search the major publications in the research area. This literature search has been performed in order to identify state-of-the-art literature to the topic of the study. In addition, the vast search for articles is done via reference databases like INSPEC, SCOPUS, COMPENDEX, IEEE Explorer, Wiley Online Library, SpringerLink, ScienceDirect and ACM Digital Library, to find articles containing related information. The searched is performed using open queries as well as keywords and combination of keywords. Google Scholar is also used to find the relevant information to use. Many articles have been found in these searches, but the focus was on information that was more relative to the subject of the thesis research. Therefore the resulting articles and website articles are evaluated as relative and reliable in terms of content, context and references by reading their abstracts, introductions and conclusion parts. The related work about this thesis research work is presented as follows.

Holm et al [72] is the most related study in which the authors suggest that the CVSS need some improvements to make it more valid and reliable. This study presents a relationship between the time needed to compromise computer systems and the vulnerabilities of these systems through various system-level vulnerability metrics using CVSS data. 34 actual compromises dataset was used for analysis which was collected during an international cyber defense exercise. More than 100 people were involved in this exercise. The authors of this study suggest that there should be a relation between system level vulnerability and time-to-compromise. This study results shows that there is no strong correlation exists between time-to-compromise and any vulnerability metric. Further the authors of this study argue that the reason behind this could be that the algorithms used in CVSS(equation 1-4) has not been validated so far and suggest that these algorithms need some improvements.

Liu and Zhang [73] have discussed CVSS in their study and some drawbacks has been identified in CVSS. For example (i) they have identified that CVSS algorithms rank too many vulnerabilities as high and too few are ranked as medium vulnerabilities. (ii) They also argue that a faulty approach is used to score the vulnerabilities. For example according to them cross-site scripting vulnerabilities should not be scored as having "partial" impact on the integrity, availability and confidentiality as they have a major impact on the end system. Such vulnerabilities in CVSS have no impact on the availability and confidentiality but have partial impact on the integrity of the system. To deal with such problems, the authors suggest a new approach to score the vulnerabilities that have been scored largely by CVSS version 2. However the formula proposed by these authors has not been checked for its validity and reliability. Only distribution of high, medium and low is analyzed in this study. Claiming that the vulnerabilities are distributed normally and this formula supports does not sound good. Why one would require having more medium vulnerabilities than low or high vulnerabilities or vice versa. It is always required to rank the vulnerabilities that are more related to the real world scenarios.

Houmb et al. [74] estimates the impact of the security vulnerabilities and the frequency of these vulnerabilities by reorganizing the CVSS metrics. The Impact in this study is defined as CVSS impact and CVSS Environmental metrics. CVSS Exploitability and CVSS temporal metrics are used to measure the frequencies. The authors propose that Bayesian Belief Network should be used to relate these metrics. This approach is only discussed theoretically and no empirical data is presented to support their claim.

Wang et al. [75] presented a formula for the CVSS Environmental and Temporal score metrics. They used a developed software to present their work. But they also not evaluated their work in order to check the validity and reliability of their work.

Gallon[76] included the CVSS environmental information to analyze the impact on the CVSS score. He changed the formula for calculating adjusted Impact score and added environmental context information to it. But again there is no empirical data to support his claim. Such data is important to evaluate the validity and reliability of such claims.

By reviewing the related work and Bearing in mind all these related studies and information about the CVSS and its improvement as mentioned above, it is concluded that CVSS has not been validated so far with empirical results. Many researchers have proposed different models for improving the CVSS but all of them provide only theoretical conclusion for such improvements. No quantitative data are available for these models to make them valid and reliable. Therefore lack of such quantitative data (empirical study) suggests that quantitative approaches, other than observation, are needed. The authors of [72] also suggest that CVSS algorithms need some improvements.

RESEARCH METHODOLOGY

This chapter presents different research methodologies available in the literature and motivates the research methodology used in this thesis research.

According to [17], the educational research has and will grow to produce new knowledge. This growth in the knowledge does not occur naturally but rather the knowledge grows through the research studies of scholars which include theorists, practitioners and empiricists - and therefore it is a function of problems posed, questions asked, observations made by those who perform the research.

Therefore the aim of any research is to discover new knowledge. Then such knowledge is presented in a research text in a way that can be easily followed through the whole text. The researcher must persuade the readers of the truthfulness and accuracy of the knowledge. The correct use of the accepted research methods can ascertain the truthfulness of such knowledge. But before going to describe the different research methodologies there are some concepts to be discussed first.

THEORY

The term theory is used in many ways but it is most commonly used to explain the observed natural phenomena. Therefore theory is the result of thinking about something in a contemplative and rational way. Almost all fields have theories that are used as analytical tools to understand, explain and predict about the subject at issue.

While the term "theory" in the modern science of today, refers to scientific theories. These scientific theories are well-established explanation of nature, consistently made with well-known scientific method, and fulfill all the criteria mentioned by modern science [18]. Such theories are communicated in a way that anyone in the field can understand them easily.

THEORY AND RESEARCH

It is not very easy to illustrate the natureof the link between research and theory. There are many issues that make it difficult, but the particular ones are:First, when one is talking about a theory, what type of theory he/she is talking about. The second issue is about the data that are collected in a research; whether the data are collected to test the theory or to present new theories [19].

RESEARCH METHODOLOGIES

Methodology in a research provides a guideline system for solving a problem in a consistent way, and is an important part of the research. This guideline system consists of different components such as techniques, methods, phases and tools for solving research problems. Therefore a research methodology is a traceable way of solving a research problem. And if correctly carried out itprovides an accurate results to the given research problem. However this does notassure that the correct research methodology is chosen or the research problem at hand is relevant, solvable or well formulated. Researchers must choose a correct methodology as well as that methodology must be carried out correctly.

As mentioned in [20] there may be different alternatives to solve particular research problems. In turn these alternativeslead to different research methods. Therefore no one can say that each research problem is solvable with only one research method. The researcher must select explicit choicesand motivate them. Relevant literature can be used to gain the knowledge about the research methods.It is not just enough to say that one is for example conducting experiments, surveys or an interview and so on. Although each one of them is well documented and well known scientific research method. A sufficient knowledge is required to use one of these choices or any other research method. Once a research method is chosen then it must be followed correctly and consistently.

When it comes to thesis writing, a research methodology can be divided into four levels, where each level is related to one another [20].

Logical level

It is the first level in this division as mention in [20] and it represents the relationship between the empirical data and theory. The first thing that needs to be dealt with in the beginning of the research is whether there is any idea or guess about the solution to the problem. If there is any idea or guess then it is called a hypothesis [20]. As an example If " X then Y" is the usual form of a hypothesis where X is an independent variable and used to manipulate to see what kind of changes occur in variable Y. Conducting a research in this way is called deductive research, where a hypothesis are empirically tested.

Deductive theory shows the most common nature of the relation between research and the theory. Based on what is known about a particular domain, the researcher deduces a hypothesis which then must be tested empirically [19]. Concepts, within the hypothesis, require to be translated into researchable questions. The researcher needs to show that how the data will be collected in relation to the concepts that are embedded in the hypothesis. The process of collecting the data is driven by the theory and the hypothesis deducted from the theory. Figure 1 shows the steps that are carried out in deductive research [19].

Theory

Hypothesis

Data Collection

Data Analysis

Findings

Confirmed or regected (hypothesis)

Revision of the theory

Figure 1: A deductive research process described in [19]

If there is no idea or guess (hypothesis) then a research is begun by collecting data (empirical). Based on the methodology and systematic analysis of this empirical data, conclusions are drawn and new theories are built. This way of doing research is known as inductive research [19]. Figure 2 represents the inductive research process.

Inductive research is a bottom-up approach that moves from specific to general. The specific means the observation/findings and based on these findings a new theory (theories) is presented.

Observation / findings

Analysis

Tentative hypothesis

Theory

Figure 2: An inductive research process

Hence it is a useful to think about the nature of the relationship between research and theory in terms of inductive and deductive strategies.

Since this thesis research is testing the CVSS artifact with the hypothesis that whether the current CVSS provides valid information for quantifying the severity of security vulnerabilities, therefore the research strategy is deductive. Empirical data will be collected to test the artifact of the CVSS.

When the researcher distinguishes the type of relation between the research and theory- deductive or inductive, then it is also very important to distinguish between two research strategies: quantitative research and qualitative research as mentioned by [17, 19, 20] and further explain that diverse communities are involved in research and on a methodological level these communities can be dividedinto two general research categories: qualitative and quantitative.

Quantitative research: This type of research methodology relates to deductive research approach to the relation between research and theory, and it testifies the existing theories. Quantitative research methodology is used in those researches where it is required to identify characteristics of a phenomenon or to examine the correlation among different phenomena [19, 20, 21]. Therefore the quantitative research methodology is related to measurable quantities and is applicable to those phenomena that can easily be represented in term of measurable quantities. Quantitative research produces statistical data through the use of survey research, using different data collection methods such as structure interviews and questionnaires and so on [22]. Figure 3 shows various steps that are carried out in quantitative research as mentioned in [19].

Theory

Hypothesis

Research Design

Research subjects/respondents

Data collection

Analyze data

Findings/conclusions

Figure 3: Quantitative Research Process

Qualitative research: This type of research methodology relates to the inductive research approach to the relation between research and theory, and emphasizes on the generation of theories. The emphasis of this type of research methodology is on words rather than quantification [19] where behavior, attitude and experiences of the participants are explored by the use of various data collection methods such as interviews or focus groups.

Figure 4 shows various steps that are carried out in qualitative research as mentioned in [19].

General Research

Questions

Selecting relevant subject

Relevant data collection

Data interpretation

Theoretical and conceptual work

Collection of further data

Findings/conclusions

Figure 4: Qualitative Research Process

Approach level:

Once it is distinguished what research strategy a research will use then there is a need for a systematic way of working to solve a research problem that comprises different data collection methods [19, 20]. A number of different approaches for research methodologies are presented in literature. For example field studies, case studies, experiments and surveys are the most common approaches that are used for research. To select an approach at this level mostly depends on the choice that is made on a logical level.

In the previous section about the logical level, a description of research strategy is presented in order to distinguish between qualitative and quantitative strategies. However by selecting any of the two research strategies will not take the research further to find a solution for the research problem. There are some other decisions to be made about the way in which the research will be carried out. These decisions are related to the choices about the research design and research methods. Since the two terms are often confused to have same meaning but there exist a difference between the two. Research design is a structured framework that guides the execution of the research method and to analyzethe subsequent data [19]. While research methods are the tools used in a research to collect data for further analysis. Different research methods are associated with different research designs. [19]mentions five different research designs that include: experimental design, longitudinal design, survey design, comparative design and case study design. But selecting a specific research design will not provide the data for analysis, therefore to collect the research data; some data collecting method will be required. These designs are briefly discussed below on by one.

Experimental design: The experimental research design manipulates independent in order to find the variation in the dependent variable to establish cause-and-effect relationships between the two variables. Experimental research answers "What if" kind of the question. The researcher sets or controls the independent variable and measures the dependent variable to find any variation pattern [19]. But the principle is, try one variable at a time and see what kind of variation happens.

Longitudinal design:A longitudinal research design is an extension of the survey research and entails repeated observations over a period of time for the same variables. In longitudinal research design, the sample is surveyed at a single point of time and the same sample is surveyed again at another point in time. Therefore the data in longitudinal research design are collected for the same sample at two different occasions [19].

Survey design:Cross-sectional design is another term used for surveys and involves collecting the data on more than one case for analysis. The survey research design is used to collect quantitative or qualitative data at a single point in time in connection with two or more variables, which is then analyzed to examine the relationship between variables and to detect variation in the patterns as association [19]. Since the data on variables are collected simultaneously (more or less), therefore there is no time ordering of the variables and the researcher does not have any control to manipulate the variables.

Comparative design:Comparative research design uses more or less identical methods to compare two or more contrasting cases. The purpose of the comparison is to better understand the relations between two or more situations or cases when they are compared [19]. This design may involve quantitative or qualitative research in order to compare the two cases. The key importance of this research design is that it allows the distinguishing characteristics of the two or more contrasting cases for theoretical reflection about contrasting findings.

Case study design: The case study research design involves the detailed and intensive analysis of a single case. A case study is defined by K. Yin [23] asa research design that uses an empirical inquiry to examine an existing phenomenon within its real-life context; when the boundaries between context and phenomenon are not clearly obvious [23].Multiple resources of evidence are used to examine such phenomenon.

After the choices made at the logical and approach level, a third level comes which is called method level.

Method level:

Data is collected in all types of researches and is considered an important aspect of a research. It is obvious that selecting a specific research design will not provide the data for analysis, therefore to collect the research data, some data collecting method will be required in order to collect the data in a systematic way. In literature different methods are described for data collection. The selection of a method for collecting data depends on the choices that are made on logical and approach level [20]. Questionnaire, interviews, observation and so on are the most commonly used data collection methods in researches. While Quantitative and qualitative research use different data collection methods.

Interviews:The research interview is the most commonly used data collection strategy. Interviews can be used in both Quantitative and qualitative research to collect the data for further analysis. Some limitations of interviews are that interviews are slow and more expensive and it can be biased by the interviewer as explained by [24].

According to [19, 25] interviews are categorized as

Structured: A structure interview is also called a standardized interview. It involves the administration of an interview by an interviewer. The purpose of this type of interview is to present all respondents with exactly the same context of questioning. This means that each respondent receives the same set of question in the same order. This type of interview is the typical form of interview with surveys research.

Semistructureinterviews:Semistructureinterview lies between structure and unstructured interviews. This type of interview uses open ended questions. Data is collected by taking notes or recording. The interview usually starts with more general questions and most of the questions are created based on the interviewee’s response during the interview.

Unstructured interview: Unstructured interview is like informal conversation where the interviewer and interviewee talks freely. The researcher may use a single question and the interviewee is then allowed to respond freely. The researcher develops questions according to what the interviewee says during the interview. This type of interview is useful when additional information is required for the research topic.

Following are different types of interview set –up that are usually used [19].

Face-to-face: This type of interview set-up enables the researcher to quickly establish rapport with a potential respondent which encourages the respondent to fully participate and cooperate. Highest response rates are obtained in survey research with this type of set-up ambiguous questions and answers are clarified by both the interviewers and respondent during this type of interview. A limitation for such interview includes impracticality whena large sample is required to investigate [26]. It is also expensive and time consuming travelling between respondents.

Telephone: This type of interview is less time consuming and less expensive as compare to face-to-face interview. One of the advantages is that the interviewer bias can be removed to significant extent.

Computer Assisted Personal Interviewing (CAPI): In this type of interview the respondent instead of completing a questionnaire directly enters the information into the database with a laptop or hand-held device. This is an easy way of answering the interview question but, however, this type of data collection method requires that both interviewers and respondent have computer and typing skills.

Questionnaire:Questionnaire is data collection method in which the respondent answers a set of questions in a predetermined order. The questionnaire is a useful method to collect a large volume of data with respondent anonymity. This method obtains a more valid and trusted responses as it provides anonymity for the respondents [19]. One other advantage of the questionnaire method is that the respondent can answer the questions without the interviewer bias or error. The main difficulty with this method is to get a high response rate as mention in [27, 28]. A follow up approach is suggested by [29] to avoid such a situation. This follow up can be a phone call or an email to the respondent for reminding about the questionnaire.

Saunders et al. [30] classifies the questionnaire as

Online questionnaire: Using electronic mail or web based questionnaire.

Postal questionnaire: questionnaire is distributed through the post, respondents complete the questionnaire and returned by post.

Delivery and collection questionnaire: Questionnaire is delivered and collected by hand.

Telephone questionnaire: This type of questionnaire is managed using the telephone.

Structured interview: Interviewer and respondent meet face to face but the interviewer does not change the questions.

Analysis level:

The fourth one is the analysis level. At this level the collected data are classified and categorize incoherent and in systematic in order to draw the conclusion. The selection of analysis method depends on the type of data collected in a research [20]. There are two types of data, one is quantitative and the other one is qualitative. Quantitative data collection requires quantitative analysis while qualitative data collection needs qualitative analysis.

THESIS RESEARCH APPROACH

This section describes the research methodology the thesis research will use to collectdata for the investigation of the research problem presented in chapter 1. This methodology will be developed from the research question discussed in chapter 1. This section will also provide a justification for the research design and strategy used by this thesis research.

The Common Vulnerability Scoring System is the de facto standard for comparing the severity of IT security vulnerabilities. Therefore it is most important to evaluate that whether CVSS provides valid and reliable information for measuring the severity of these vulnerabilities are not. Validity is related to the most important information that CVSS contains for the purpose quantifying the severity of the security vulnerabilities, while reliability is related to the completeness of the algorithm (mentioned in the CVSS introduction chapter ) used for calculating the severity of the security vulnerabilities.

As described in chapter 1, this thesis research is testing the CVSS artifact with the hypothesis that whether the current CVSS provides valid information for quantifying the severity of securityvulnerabilities, therefore the relationship between research and theory is deductive. Empirical data will be collected to test the artifact of the CVSS.

After distinguishing the type of relation between the research and theory- deductive, then it is also very important to distinguish between two research strategies i-e quantitative research and qualitative researchas mentioned in above section 2.3.It all depends on the research question that what kind of data is to be collected. The thesisresearch questions described in chapter 1 show that this thesis research will collect both kinds of data;quantitative and qualitative hence a combination of both researchstrategies will be used in this research.

Once a researchstrategy is selected, then it comes to research design because there is a need for a systematic way of workingto solve a research problem that comprises different data collection methods. As mentioned in section 1.2 that expert judgment is needed for the validity and reliability of the current CVSS Scoring System. To obtain experts’ judgment and their suggestions, this thesis research will use an online survey research design where a web-based questionnaire will be used as a data collection method to collect the data from the participants. The justification for online research design is provided in section 2.5. It is very important to keep in mind that the participants (the respondents) should be security experts (see section 2.5.3 for experts sampling out of the whole population) in order to get a very reliable response. It is also very important to measure their (respondents) expertise as experts may have different responses for different vulnerabilities.

ONLINE SURVEY

There is no doubt that the internet and online communication have great implications for the research within many fields. Internet users increase every day. According to Internet World Stats [31]2.3 billion internet users were using the internetin the first quarter of 2012.  The use of internet is highly among the computer professional [19]. The internet offers a several opportunities to conduct the research, and one of them is the online survey research. There are two important distinctions as mention by [19] the researcher should keep in mind when researching via internet; the distinction between web-based and communication-basedresearch methods. The web-based method is used to collect the data through a web, for example a questionnaire is placed on the web page and then the respondent completes such a question on the web page where the data is entered into a database. Whereas the communication-based method is used to collect the data through email or similar communication medium from which the data collection instrument is launched. A combination of the two distinctions can also be used, for example, an email is sent to the respondent with the link to the web page where the survey questionnaire is placed. The respondent simply opens the link to the web page either by clicking on it or copy and pastes it in the browser and completes the questionnaire.

Two forms of the online surveys have been used very commonly since 1986 [32]. Email-based which is an asynchronous type of survey and web-based which is a synchronous type of survey. Web-based survey provides some advantages over the email-based survey. Web-based provides a user interface for the participants where the participants complete the questionnaire and the data entered by the participants are stored automatically in a database system. While in the email-based survey, the questionnaire is sent either embedded in the email message or as a document attached to the email message. The participant completes the survey and sends the questionnaire back to the surveyor. Then the surveyor manually enters all the data into the database system for further analysis.

Therefore online survey is the best way to collect the large volume of data simultaneously at a point in time. It is the systematic way of collecting the data on the subject at issue by asking questions from the participant and then analyzes the data to generalize the results to the whole population represented by the sample of the participants [33].

SURVEY CHALLENGES AND OPPORTUNITIES:

The online survey’s greatest strength lies in its versatility. All types of information can be collected by asking questions of others [34, 35]. The literature on online survey methodology shows many issues such as instrument design, quality of survey data, appropriateness ,validity and response rate [36,37, 38].According to [39] online surveys provide an effective way to collect information from the participants when the population is well identifies, when no probability sampling of the population is essential, and when the respondents have the necessary computer skills and have access to the internet. Online surveys are the fast data collecting tool and when a large volume of data is required to collect from a sampling population that is spread widely, the online survey provides an efficient and fast way to achieve such a goal. As mentioned in [30], the best way to conduct the online survey is that first send an email to the respondent and from where the respondent is directed to a web-based questionnaire to answer the questions. There are evidences that indicate that respondents provide a more valid answer to the question in an online survey than on paper based survey [40]. An expensive mechanism is provided by the technology for conducting online surveys than traditional approaches (postal or telephone). [41,42]. That is why Online surveys in many disciplines are becoming increasingly common [32,43], and many studies show that there is no difference in the results for online survey content as compare to the results for postal survey content, yet online surveys provide fast distribution [32, 44,45].

However except the strength that online survey has, there are some weaknesses that exit with the online survey. The weakness that exits in surveys is the about the quality of collected information which depends on the ability of the respondent and the willingness to cooperate (34, 35, 46). Internet access and comfort level of the respondent in filling the questionnaire have a negative effect on the online survey. There are software issues for example JavaScript, multiple platform or other plug-ins and spam blockers can turn the online survey into waste.

RESPONSE RATES

One of the important concerns of the researcher who conduct the online survey is the low response rate [47]. Therefore the value of the research depends on the participants who take part in the research by answering the online survey questionnaire. The percentage of those participants who are contacted in online survey but does not respond is called the non-response rate [38]. According to [38] there are some well-known techniques that can facilitate and increase the response rate of the participant. Table 1 represents some of these techniques.

Table : Techniques that increase response rate

Techniques for facilitatingresponse rate

Description

Participants prenotification

The participant can be personally prenotifiedabout the survey.

Survey publicity

Inform the respondents about the survey, its purpose and how the results of the survey will be used

Carefully design survey

The physical design of the survey should be done carefully. Fonts, color, layout etc. should be designed properly. Easy to read, pleasing to the eyes and so on.

Incentives

Incentives, if there is one, should be provided to the respondents where appropriate. The results of the research, if possible, should be provided as an incentive to make the respondents’ response rate high.

Survey length

The length of the survey should be managed in a proper way.

Reminder notes

The respondents should be reminded in a week about the survey after the distribution of the survey. This can be achieved either by sending a reminder email or by phone (if possible).

POPULATION AND SAMPLING

Population:A universe of units from which the sample is selected for investigation to estimate the whole population.

Sampling: sampling is a segment of the whole population which is selected for investigation to estimate the whole population. It is the subset of the population under research investigation.

It is not possible and unlikely to be able to send the questionnaire to the whole population. Similarly it is also unlikely to be able to interview a large number of participants, especially when they are geographically distributed. Conducting survey research of interviews is more expensive and time consuming in such situations [19]. Therefore it is very important for the researcher to sample the survey participant from the total population. This thesis work will also consider matters relating to the sampling in relation to survey research involving the data collection by web based questionnaire. According to [19], sampling of survey research is the key step in the research process. Personal judgment, prospective respondents’ availability and implicit criteria can influence the sampling process [19]. So such limitation simply means a biased sample and would not represent the population from which the sample is selected. Therefore some steps should be followed to keep such bias to minimum level. Based on the literature, there are many methods to sample the population for example the following are the most commonly used ones.

As explained in [48] that there are many sampling methods available for internet based survey (email-based or web-based) that can be used to sample the population. Sampling methods are divided into two broad categories: probabilities based sampling and non-probabilities based sampling. These internet based survey and the corresponding associated sampling methods are provided in the table 2 and table 3.

PROBABILITY SAMPLING

The most basic form of probability sample is simple random sample. Each unit of the population has equal probability to be selected as a sample for further investigation in a research. The most important aspect about this process is that there is no human bias present to influence the sampling process. The importance of the probability sampling in a survey research is that it is possible to estimate the whole population based data collected from the random sample [19]. Therefore findings from a sample, using the random sampling procedure, can be generalized to the population. Population data and sample data should not be treated as the same but the researcher can make inferences from the information about the sample to the population from which it is selected.

Table : Probability-based sampling

Probability-based Methods for sampling

Email-based survey

Web-based survey

A list-based sampling frame

YES

YES

Non-list-based survey using random sampling

YES

YES

Intercept based surveys

-

YES

Pre-recruited panel survey

YES

YES

A list-based sampling frame:

The internet based surveys which uses a list-based sampling framecan be conducted like a traditional survey where the simple random sampling is implemented in a straightforward way. It only requires a list of contacts (an email list of the participants for internet survey). As shown in table 2 that this sampling method can be used with both email and web based surveys. This kind of the sampling method is largely applicable to the homogenous groups for which the email list as a sampling frame can be collected for example government organization, universities, companies etc... [48]. Multi-stage and cluster, more complicated sampling methods, are very difficult or even impractical to be used for the internet surveys [48]. According to [48, 49] multi-stage sampling method takes a lot of efforts to assemble the data and then to conduct the internet-based surveys.

Non-list-based survey using random sampling

It is a sampling method used for sampling the population based on the probability without considering any sampling frame. In traditional survey approach, telephone based survey uses random digit dialing (RDD) which is a non-list-based random sampling method but in internet-based surveys there is no such equivalent exits of RDD. Practically speaking it is not possible to randomly generate email list. However these sampling methods can be used in internet-based survey by contacting the participants through other traditional means for example telephone. But this will introduce some other complication and costs [48].

Intercept based surveys

As explained in [48] intercept surveys are popup surveys on the web page or web site. Every kth visitor to a web site or web page is systematically sampled using this sampling method. These kind of surveys are useful for marketing purpose where the customer satisfaction is surveyed.

Pre-recruited panel survey

This type of survey is generally related to groups of individuals already agreed to participate in the survey. Individuals in this type of survey are contacted via different means other than email or web for example through telephone or postal mail to participate in the internet-based survey that requires probability samples. For example many companies have panels of individuals who are pre-recruited using the probability-based sampling method. These panels can be further sub-sampled according to the researcher’s requirements. Then the researcher can use these pre-recruited groups of individual in the internet-based survey [50].

NON-PROBABILITY SAMPLING

Non-probability is an umbrella like a term that is used for all forms of sampling that are not selected according to any probability approach. Convenience sampling, snow ball sampling and quota sampling are the different procedures used as non-probability sampling.

Table : Non-probability based sampling

Non- probability-based Methods for sampling

Email-based survey

Web-based survey

Unrestricted self-selected surveys

-

YES

Volunteer panel survey

-

YES

Harvested email list survey

YES

YES

Unrestricted self-selected surveys

As explained in [48] these types of surveys are open to public and anyone can take part in the research by participating in the survey. Web sites or web pages are used to post these surveys and anyone, browsing through the web site, can choose to participate in the survey. Different media can be used to publicize the survey for example internet-bas