Collaborative Intrusion Detection And Signature Based Alert

Published Date: 02 Nov 2017

Abstract

Current reactive and individual network security products are not capable of withstanding the onslaught of diversified network threats. As a result, a new security paradigm, where integrated security devices or systems collaborate closely to achieve enhanced protection and provide multi-layer defenses is emerging. In this paper, we present the design of a collaborative architecture for multiple intrusion detection systems to employ mutually to detect network intrusions at real time. The detection is made more efficient and effective by using collaborative smart agents, relevant knowledge base and combination of multiple detection sensors. The architecture is composed of three parts: Collaborative Alert Indexing, Signature based Alert Estimation and Alert Correlation. The architecture is meant at dropping the alert excess by correlating outcomes from numerous sensors to produce condensed views, dropping false positives by combining host system and network information into the estimation technique and correlating measures based on reasonable relations to produce global and manufactured alert report. The architecture is designed as a layer above intrusion detection for post detection alert analysis and security actions. The architecture has been executed and the executed results are presented in this paper.

Keywords: Network security; Intrusion detection; Alert; Smart agents; Collaborative work

1. Introduction

Network computing has become an integral part of our daily life with the emergence of the digital society. With the advances in Collaborative work techniques, network based collaboration among individuals and groups can dramatically boost our productivity. But on the other hand, with the advent of the Internet Era, the widespread reliance on network as well as Collaborative work technology makes computer attacks launched over the Network more devastating than ever before. Due to its distributed nature, a distributed Collaborative work application can be more vulnerable than traditional individual applications. For instance, intruders can easily exploit a Collaborative work application by disguising themselves as remote trusted entities and sending an executable file to the victim and executing it via a shared application. Consequently, the success of a Collaborative work application will be largely determined by the success of its security systems. Therefore, security systems inevitably become an essential part of Collaborative work applications. On the other hand, in response to the daunting threats of network attacks especially the increasing amount of distributed, coordinated malicious attacks such as DDoS attacks, the collaboration among different security systems, security information as well as the collaboration among experts from all over the world are becoming crucial in winning this war against spiteful network hackers. Consequently, now-a-days Collaborative work techniques are playing an important role in designing, developing and deploying security systems, devices and policies.

In response to the challenges from malicious network attacks, a promising approach to deterring intruders is to detect, record and analyze attacks and finally prosecute spiteful attackers using the evidence collected through the above process. Computer and network forensics is such an approach. This newly emerging discipline is becoming more important as the society is recognizing the seriousness of network attacks. It involves capturing, recording and analyzing network events in order to discover the source of security attacks or other problem incidents. It attempts to deter hackers from attacking a system as the chance of their discovery increases. It also searches for evidence after an attack has occurred. This evidence will be useful in prosecuting the hackers or in devising counter-measures.

Intrusion Detection System (IDS) is an essential part of computer and network examining. Intrusion detection is a kind of security device or system that is deployed to observe network and host behavior including data flows and information accesses etc. and discover suspicious behavior and capture relevant evidence for later use. Its main purpose is to detect real-time, ongoing intrusions and warn system administrators through alarms so that actions could be taken to stop the intrusions or discover the damage if the attacks had already succeeded.

Currently, there are two basic approaches to detection of an intrusion [1]. The first approach, called the anomaly detection (also called the behavioral detection), is to define and characterize the correct static form and the adequate vibrant activities of the system and then to detect illegal changes or unfair behavior. The second approach, called the misuse detection (also called the signature detection), involves distinguishing known ways to break into a system.

Every known penetration technique is typically depicted as a prototype. The misuse detection system looks for explicit prototypes. The prototype may be a fixed bit string such as a precise virus bit string inclusion. Instead, the prototype may describe a suspect set or sequence of actions.

Intrusion detection systems (IDS) have been built in the past 15–20 years using both the approaches: the anomaly detection and the misuse detection. Some of the IDS systems including EMERALD [2] and DIDS [3], are exploring a hybrid approach. In recent years, intrusion detection products have been widely deployed and are beginning to gain acceptance as a worthwhile investment. But neither of these two detection systems is quite satisfactory. For example, the anomaly detection systems often generate too many false positives. This happens because the deviation from the normal behavior does not always correspond to the occurrence of an attack. Besides, the critical threshold is hard to define precisely. The misuse detection systems can only detect those intrusions depicted in their signature repository. Therefore, new or slightly modified intrusions cannot be captured by misuse based intrusion detection systems.

In addition to the above weaknesses, IDS products are subjected to many other problems including alert flooding, too many false positive and false negative alerts, isolated alerts against a series of attacks, blindness to network and hosts they are observing etc.

Many of the above weaknesses in traditional IDS products are due to the lack of various collaborations including: (a) collaboration among different detection systems, (b) collaboration between intrusion detection and other network management operations, (c) collaboration between detection and other security systems. In order to overcome this deficiency, in this paper, we propose an architecture to enable collaboration among multiple intrusion detection systems using distributed smart agents and Collaborative work techniques. The ultimate goal of the proposed collaborative architecture is to make intrusion detection more accurate, more efficient, and easier to use by system administrators. Where appropriate, some of these processes may even be automated. The key to the realization of this goal is the use of a collaborative architecture that uses distributed smart agents and relevant information knowledge bases. In the proposed architecture, different IDS products and the protected hosts and network asset information are integrated as a holistic security system through the deployment of a distributed system of autonomous smart agents interacting via message exchange and thus making decisions in a cooperative and coordinated manner. The architecture sits as a layer above intrusion detection system for post-detection alert analysis and security actions.

2. Related work

Realizing the limitations of single detection systems, researchers began to explore the benefits of collaboration among different IDS products. The main objective of the IDS cooperation is to reduce the number of alerts generated by correlating different IDS outputs and discard false alerts. By threading multiple alerts generated by related attacks, cooperating IDS modules will be able to provide a global view of intrusion behavior.

The first IDS collaboration research was initiated in the IDES [4] project and then refined in the EMERALD [2] project. Currently, there are a number of ongoing projects in the area of alert indexing and the false positive reduction.

Honeywell is developing Argus, a qualitative Bayesian estimation technology to combine results from multiple intrusion detection systems [5]. Cuppens [6,7] is developing an intrusion detection indexing and correlation module (MIRADOR) using snort and e-trust. An expert-system based approach for similarity formulation is used in their work. SRI international [8] is using a probability-based approach to attribute similarity recognition. Another approach is the Tivoli Enterprise suggested by Debar and Wespi [9]. Their correlation technique uses a significant system to state what types of alerts may pursue a given alert category.

At the Concurrent Engineering Research Center (CERC) of West Virginia University [10] and Illinois Wesleyan University, we are developing a generic collaborative alert management and analysis architecture for multiple IDS products with smart agents and Signature based alert estimation. Our approach differs from the above ones in the following aspects. First, we categorize the correlation into two different types: (a) information asset correlation and (b) the alert correlation. Information asset correlation uses the network and host hardware and software information to estimate the alert priority and the likelihood of the success of the attack. The alert correlation is to correlate different alerts based on the logical relations among them to provide a global vision of the effects of intrusion. Second, we designed a generic collaborative architecture for multiple diversified IDS and alert estimation. This architecture sits as a layer above IDS products. Third, based on the specific alerts, appropriate security solutions collected from multiple vulnerability monitoring corporations and organizations will be provided with the alert in order to assist security administration in taking the appropriate actions. Although this architecture design that uses smart agents and relevant knowledge base to provide a layer above intrusion detection is novel, using agents in detecting malicious intrusions itself is not new. It has been explored by many researchers.

Distributed Intrusion Detection System (DIDS) [3] uses agents to aggregate audit reports from hosts. The architecture consists of a host manager, an observing process or collection of procedures; Autonomous Agents For Intrusion Detection (AAFID) [11] is a distributed anomaly detection system that employs autonomous agents at the lowest level for data collection and analysis. At the higher levels, other agent entities are then used to obtain a global view of behavior; Helmer et al. [12] use mobile lightweight agents in intrusion detection. Their agents travel between monitored systems to obtain, classify and correlate information. Finally, this information is used to detect suspicious behavior.

All the above researches use agents to some extent to collect information for the purpose of detection. Therefore, the information is often retrieved from audit logs etc.

Besides, in many agent based detection approaches such as DIDS, agent technology is used only minimally. Our design is a layer above intrusion detection. The information is mainly used for the alert estimation and security decision making. We also exploit agent technologies such as agent communication languages and services for the collaboration among distributed smart agents. The architecture design we have been developing is dubbed as Multi level architecture.

The implication is that the third eye is the system we are developing to watch over a computer network.

3. The Multi Level architecture design

As shown in Fig. 1, the collaborative architecture consists of three main components: (a) collaborative alert indexing, (b) Signature based alert estimation and (c) alert correlation. Each of these three parts is depicted in the following sections.

3.1. Collaborative alert indexing

Collaborative alert indexing is mainly aimed at mitigating alert flooding. Meanwhile, a loosely coupled collaboration among multiple IDS products is also achieved by using smart agents and advanced algorithms. This component provides three functions: (i) alert preprocessing, (ii) alert clustering and (iii) collaborative alert merging.

These functions are provided by the following distributed agents.

3.1.1. IDMEF agents

To aggregate alerts from multiple IDS products with different output formats, first the component needs to convert the diversified formats into a unified standard representation. The format we chose is the Intrusion Detection Message Exchange Format (IDMEF) [13], which is emerging as an industry standard.

The conversion of heterogeneous alert output into the standard IDMEF format is performed by IDMEF agents. An IDMEF agent is an independently running entity that collects alerts emanated from an IDS product and converts them into the standard IDMEF format. In our design, for each IDS product, an IDMEF agent is developed and deployed close to that IDS product. The IDMEF agent is then dedicated to that IDS product and converts its output into IDMEF format. Therefore, an IDMEF agent can be implemented in any language and deployed in any system atmosphere provided the output is in the standard IDMEF format. Whenever a new IDS is integrated into the system, we only need to provide a dedicated IDMEF agent to serve it. Therefore, the architecture is highly scalable and can evolve with advances in IDS systems. IDMEF agents do not communicate directly with each other. Instead, they store all their outputs in an IDMEF database for later analysis.

3.1.2. Clustering agent

As IDMEF agents store alerts in the IDMEF database, another kind of agent called clustering agent is fulfilling its responsibilities. The Clustering agent groups the alerts into different clusters according to their source, target, time and classification. Each cluster is then used by a ‘representative alert’ or Meta alert to represent that cluster. Clustering is used to eliminate duplicate alerts and group alerts from the occurrence of the same attack. After clustering, the amount of alerts can be significantly reduced. Clusters are then used in alert merging process by a Merging Agent.

3.1.3. Merging agent

Currently, the collaboration among multiple IDS products in Multi Level architecture may be characterized as loose and it is transparent. We have designed the Multi Level architecture as a layer above IDS products. In the architecture, IDS products are integrated into it as black boxes. The collaboration is then achieved by this upper layer. In practice, when IDS vendors develop their own IDS products, they seldom have the collaboration with others in mind. Consequently, their alert formats are heterogeneous and detection systems are proprietary and not released to public. Therefore, direct collaboration among multiple IDS products from different vendors is very difficult and may not be feasible at all. The collaboration among multiple IDS products is done indirectly by the alert merging agent in the Multi Level architecture. The analysis of alerts from multiple IDS products results in useful information that could be gathered collectively and resolution of conflicts is all carried out by this kind of alert merging agent. This agent is also used to merge the alerts from different IDS products into synthesized alerts. Alert merging agent is highly smart because it uses a voting algorithm to solve conflicts when IDS products have different ‘voices’. The algorithm works as follows: if the number of IDS reporting alerts is greater than the number of non-active IDS systems when an event occurs, the voting algorithm will notify that an attack has occurred and generate a synthetic alert. But when only two IDS are deployed, we chose a priority based voting algorithm. When conflict arises, the algorithm will generate a synthesized alert only when the priority levels of the generated alert pass the associated priority screening levels. The priority of the synthesized alert will still be reduced one level lower compared to its original one.