Security Incident Handling Service

Published Date: 02 Mar 2018

EXECUTIVE SUMMARY

1 INTRODUCTION

Expect the unexpected. As soon as a crisis erupts, it should be immediately handled to reduce its potential impact on critical business operations. Such undesirable incidents occur unanticipated and when they do take place, damage or harm is the result. In most aspects of life, it is better to stop something disastrous happening than it is to deal with it after it has happened and IT security is no exception. If possible, security incidents should be dealt accordingly from occurring in the first place. Yet, it is unachievable to prevent security incidents. When an incident does happen, its impact needs to be brought down to adequate recommended level. Security incident handling outlines the actions to follow in an event that an electronic information system is compromised. An event is declared an incident when the confidentiality, integrity or availability (CIA) elements of a system is compromised. Significant commodities such as information and knowledge must be safeguarded at all costs. Communications within an organization and its interactions to its customer base are regarded as the life blood in this IT intensive fast paced world. If an organization is inoperative for any period of time, it may cost millions in lost business or loss of reputation. Size of an organization does not matter. Unexpected downtime influences organizations of all sizes impacting revenue, customer satisfaction and overall production. It is vital that they quickly recover from such downtime and restore operation and re-establish their presence to ensure survival. Consequently, many firms have realized the importance of setting up incident handling procedures. One of the drawbacks is that many organizations learn how to respond to security incidents only after suffering from them. In the course of time, incidents often become much more costly. Proper incident response should be an integral part of the overall security policy and risk mitigation strategy. Incident handling procedures that are in place in an organization improves to maintain the business continuity of critical operations. In today's competitive economy, a company can't afford to cease critical business operations and remain idle for long period of time because of lack of incident handing procedures. Thus, an organization needs to be well prepared for continuity or recovery of systems. This typically requires a considerable investment of time and money with the aim of ensuring minimal losses in the event of a disruptive event. The goal of setting up incident handling procedures is to know exactly what to do when an incident breaks out. This means anticipating scenarios before they occur and making appropriate decisions about them in advance. Those assessments typically demand consultation and senior management support, hence these people are needed early immediately after an incident has been confirmed. For example, just deciding who to tell when an incident occurs can be hard to determine. Management needs to provide input to respond quickly and this embarks into issues like after hours support and mixed project/support roles. External support may also be sought, resulting in additional cost, time and effort to select partners.

1.1 PURPOSE OF THE DOCUMENT

This document provides guidance to identify and record the nature and scope of a computer security incident handling service. This paper discusses the functions that support the service, how those functions interrelate and the tools, procedures and roles necessary to implement the service. It also concentrates on incident analysis. For example, we can make a comparison between a fire that broke off in an apartment and a computer security incident that happened in an organization. Similarly as a fire department will investigate a fire to know where it originated from, a Computer Security Incident Response Team (CSIRT) tries to figure out how the security incident occurred. Both the fire department and CSIRT operate in the same approach. A fire department needs to get along with other fire departments on it can depend on for additional support in peak times or to tackle a serious catastrophe. It must cooperate with other emergency units to react promptly and provide law enforcement. This document will discuss how CSIRTs interact with other organizations, such as the department that reported the security incident to it, other CSIRTs, law enforcement and the media. Both fire department and CSIRT need to properly handle information, some of which is sensitive and relevant to the individual held responsible for the crime. Information handling is considered to be an indispensable discussion subject in this paper. CSIRTs propose client confidentiality in the same manner that many emergency units do, safeguarding reporters and victims from public disclosure. CSIRT survival depends on handling confidential information appropriately, because if it can't be trusted, nobody will report to it, thus making it almost useless. CSIRTs have committed permanent staff as well as part-time, volunteer staff and reliable security experts to handle an unexpected security emergency. Its staff is at the frontline in event of a crisis, CSIRT achievement depends on their interaction with the outside world and the image that they project by the way of performing their duties and the service quality that they provide. To attain such high level of success, recruiting suitably competent staff seems to be a complicated process. People in charge of appointing CSIRT staff mistakenly look for unsuitable set of talent and ability in prospective employees. For that reason, this paper discusses staffing and hiring concerns and actions to guarantee that CSIRT staff offer reliable, pleasant and specialized service. Other services besides the incident handling service, such as the supply of intrusion detection assistance and vulnerability handling are also provided by CSIRT. The information in this paper is understandable in such a manner that is basic to the reader to put it into operation to any type of CSIRT setting, from in-house team for a company to an international coordination center. This document is intended to present a valuable foundation to both recently created teams and existing teams where there is a lack of clearly defined or documented services, policies and procedures. This paper is more appropriate to use during the early stages when a company has acquired management support and funding to set up a CSIRT, before the team becomes operational. Moreover, this paper can be still a valuable reference document for already operational teams.

1.2 INTENDED AUDIENCE

The general CSIRT community who may require a better knowledge of the composition and objectives of their existing teams will benefit from this document. It also targets individuals and organizations who are likely to join the CSIRT community in the near future. It is precisely aimed at managers and other personnel who take part in the process of setting up and leading a CSIRT or managing incident crisis. The list may include

• Chief Information Officers, Chief Security Officers and Information Systems SecurityOfficers
• Project leaders and members in charge of creating the team
• CSIRT managers
• CSIRT staff
• IT managers [1]

Higher management levels and all CSIRT staff can use this paper as a useful reference. This document can also be utilized by other individuals who work together with CSIRTs. This may include members of the

• CSIRT constituency
• law enforcement community
• systems and network administrator community
• CSIRT parent organization or other departments within the parent organization such as legal, media or public relations, human resources, audits and risk management & investigations and crisis management [2]

2 MAIN CONTENT

Definition of Security Incident

The Information Security Management Handbook defines an incident as "any unexpected action that has an immediate or potential effect on the organization" [3]. Whenever the safety and stability of an information system is compromised, such instance can be referred to as a security incident. There are several different definitions of security incidents; one is "A violation or imminent threat of violation of computer security policies, acceptable use policies, or standard computer security practices" [4], another definition describes the security incident as "any event that may threaten or compromise the security, operation or integrity of computing resources" [5]. In other words, a security incident is a state of violation of security policy in an organization and the security of their information system. Security incident refers to a common term that encompasses any type of security breach regardless of location, the level of the threat or the magnitude of it. The commonly known factors of security incidents are events and actions that expose one or more basic elements of information security: confidentiality, integrity and availability (CIA) of information systems. An incident can be caused by authorized or unauthorized personnel, process, hardware or software. It can be an accident as well as a planned malicious action.

Handling security incidents

In the course of a crisis, time runs short in terms of about what to do, who will do it or how it will get done, therefore it is vital to arrange for a response in advance. The better prepared you are for an incident, the more likely you are to respond correctly. Proper set-up of an incident handling procedure can help to lessen impact of undesirable incidents. The objective of such procedure in place is to provide a framework for an orderly, coordinated response by appropriate resources within the organization. It is in a company's own benefit that it establishes a Computer Security Response Capability, a process that provides centralized response and reporting functions for security incidents. According to (Computer Security Incident Handling Guide, National Institute of Standards and Technology, March 2008), establishing an incident response capability should include the following actions:

• Creating an incident response policy plan
• Developing procedures for performing incident handling and reporting, based on the incident response policy
• Setting guidelines for communicating with outside parties regarding incidents
• Selecting a team structure and staffing model
• Establishing relationships between the incident response team and other groups,
• Determining what services the incident response team should provide
• Staffing and training the incident response team

The “Cyberthreat Response and Reporting Guidelines” report, jointly approved by the FBI and US Secret Service recommends that the better equipped a company is in the event of a security event, the better probability it has to reduce the impact of the crisis. This recommendation is actually one of the chief responsibilities of a CSIRT, to be well organized to successfully cope with an incident when they happen and to help prevent incidents from occurring in the first place. As a starting point, the team should have a strategy plan for incident handling. This plan should be supported with documented policies and procedures. According to (State of the Practice of Computer Security Incident Response Teams, October 2003), the incident response plan identifies the mission and goals of the team, the team roles and responsibilities; the services provided; and policies, procedures, processes, and guidelines related to incident handling. The incident response plan is not only intended for CSIRT employees, but also for community that they serve. From that viewpoint, both parties should be proficient about what to report, how to report it and to whom it should be reported. The plan should also describe the expected level of service that is reasonable. Staff who is accustomed with computer security incidents recognize the fact that these incidents vary in shape and size. Some are quite uncomplicated, easy to cope with and mitigate while other are extremely severe and very complicated or can have harsh impact on IT systems and necessitate proper authority to respond to effectively. In the event of a crisis, adhering to the plan in place will facilitate the organization to promptly isolate disruption cropping up on IT systems or networks as well as to assist to counteract to such events. It may alleviate potential risk such as loss of company reputation, trust or financial status. For existing CSIRTs who don't have a robust plan, they can still manage with some basic guidelines. They can make use of their current incident handling procedures as a guideline, in the meantime they can revise their existing documentation. They can rely on those basic guidelines namely the plan to handle incidents, areas of responsibility, general and specific procedures. Other typical guidelines can include an incident response checklist as well as procedures for what type of activity to report and how that information should be reported. A company needs to take into consideration several factors prior to planning an incident response capability. They include

• introducing a point of contact for reporting incidents
• pinpointing the aims and objectives of the team
• distinguishing and selecting the staff and necessary expertise
• offering direction for reporting and handling incident reports
• allocating proper security awareness and incident response training for CSIRT staff
• launching and promoting specific incident handling and security policies and procedures for the CSIRT
• exposing lessons learned with other colleagues
• designing a benchmark to monitor the effectiveness of the CSIRT
• devising strategy to allow coordination between the CSIRT and internal and external parties

Organizations or the team typically approve policies and record them. It is crucial to know what these policies consist of and to ensure that they are properly implementable, enforceable in the workplace. Like the mission statement, senior management approves and enforces policies. The policies need to be openly expressed and well understood by each team member, technical, management or administrative. It will be a difficult task for the staff to appropriately execute and carry out their duties without a clear understanding of the policy. In order to write a clear policy, it is best to avoid excessive jargon. Whenever possible, consult someone who is not in security or IT to examine the policies. Rephrase the policies if not understood. Use very short sentences. A good policy is a short one. A security policy should be concise, well segregated between the management aspect (the policy) and the operational aspect (the procedures). Moreover, a policy must be both implementable and enforceable, or else it doesn't have any purpose. It is easier to implement a policy if it is well designed and relevant to the needs and goals of the CSIRT. Truly effective policies address genuine needs within a business, making the staff willing and even eager to implement them because they make operations smoother and give the business added reliability. Top management should execute appropriate actions or steps to enforce a policy. Policies must be enforceable; otherwise they are of little or no value. Usually when a policy ismplementable, it is normally also enforceable unless it contradicts itself. Concrete measures are needed to assess the usage of the policy. Example: An example of a contradictory policy is the security policy that ranks internal information security as priority number 1 but at the same time ensures absolute privacy for its staff; the latter makes it hard or even impossible to enforce security in case of an insider threat. To successfully develop and implement security policies, top management needs to be involved in and strongly support the project (Lam, 2005). A proposal with a report of external and internal requirements and a draft assessing budget can easily persuade managers to support the development and implementation of a security project. Having management support and authorization can resolve money and time issues. These managers can allocate the required budget and allow sufficient time for development and implementation. In addition, top management has power to affect processes by requiring employees to participate (Kearns & Sabherwal, 2006).

How to Implement Security Policies Successfully

The implementation phase probably is the hardest phase in the life cycle of developing and maintaining security policies. Many organizations fail in this phase. To effectively and efficiently implementing security policies, teams first need to resolve many issues. Lack of strong management support (Fedor et al., 2003; Lam, 2005), lack of budget (Kearns & Sabherwal, 2006; Martin, Pearson, & Furumo, 2007), lack of implementation time (Walker & Cavanaugh, 1998), lack of strong leadership (Fedor et al., 2003), lack of awareness of benefits of implementing security policies—“why for” (Hansche, Berti, & Hare, 2004)—, or ineffective communication with users (Jackson, Chow, & Leitch, 1997; Walker & Cavanaugh, 1998) may cause problems. Resolving all of the above issues can help in successfully implementing security policies.

Computer Security Incident Response Team (CSIRT)

A team is a focal component of incident response plan, policy and procedure creation so that incident response is dealt effectively, efficiently and consistently. The team should cooperate with other teams within the organization towards a central goal which encompasses the plan, policies and procedures. Outside parties such as law enforcement, the media and other incident response organizations can also be contacted. Computer Security Incident Response Team is regarded as the nerve center of an incident response plan. It is normally composed of a team manager, a management advisory board and other permanent and temporary team members. The temporary staff provides advice on technical, business, legal or administrative issues, depending on the nature and scope of the incident. The team assists the organization to identify and document the nature and scope of a computer security incident handling service. The team manager supervises labour of the team members, presents ongoing status information to the Chief Information Officer (CIO) and other senior management and requests assistance on expert advice outside of IT department when needed. This role leader should be accustomed with computer security issues, the function of IT areas and staff, general company operations as well as the duty of other employees in the institution who may serve as resources for the CSIRT. Under challenging situations, the team manager must be able to coordinate teamwork with other staff and to deal properly with circumstances that necessitate discretion or confidentiality. The technical leader's role is to assess the characteristics and severity of an incident, propose recommendations on security control and recovery issues to the team manager and requests on additional technical resources if needed. This role should possess a broad understanding of operational and systems security. Other employees can join the team on a spontaneous basis and remain team members until closure of incident. Additional resources may be required to serve areas such as: law enforcement, legal, audit, human resources, public relations, facilities management or IT technical specialties. The table below shows a list of members who should be included in the CSIRT and their roles in the team.

Table 1: Team members in IRT

Source: table from page 4-2 of Incident Response Procedure for Account Compromise Version 1.2 2004 by Visa International Besides their technical expertise, CSIRT staff distinctive quality is their motivation and talent to stick to procedures and to present a professional image to customers and other parties working together with them. In other works, it is more convenient to appoint staff with less technical expertise and excellent interpersonal and communication skills and subsequently train them in a CSIRT-specific environment than vice versa. Communication of a team member who is a technical expert but has poor communication skills may brutally ruin the team's reputation while interactions that are dealt with competently will assist to improve the team's standing as a valued service provider. Possessing a broad range of interpersonal skills is significant since team members are frequently in contact with each other and other parties such as law enforcement, legal, human resources. Thus, these professional interactions that CSIRT employees adopt will influence the reputation of the team and special concern to an individual's interpersonal skills matters. Some interpersonal skills, required for incident handling staff, are listed below:

• logical judgment to formulate effective and suitable decisions in time of crisis or under pressure or strict time constraints
• effective oral and written communication skills for interaction with other parties
• discretion when dealing with the media
• aptitude to follow policies and procedures
• enthusiasm to learn new things
• challenge to work under pressure
• teamwork
• reliability to maintain team's reputation and status
• readiness to accept one's own mistakes
• problem solving skills to efficiently handle incidents
• time management skills for high priority tasks

Apart from interpersonal skills, CSIRT staff should possess fundamental understanding of technology and issues on which they base their expertise. The following technical know-how is crucial for CSIRT staff:

• public data networks (telephone, ISDN, X.25, PBX, ATM, frame relay)
• the Internet (aspects ranging from architecture and history to future and philosophy)
• network protocols (IP, ICMP, TCP, UDP)
• network infrastructure elements (router, DNS, mail server)
• network applications, services and related protocols (SMTP, HTTP, HTTPS, FTP, TELNET, SSH, IMAP, POP3)
• basic security principles
• risks and threats to computers and networks
• security vulnerabilities/weakness and related attacks (IP spoofing, Internet sniffers, denial of service attacks and computer viruses)
• network security issues (firewalls and virtual private networks)
• encryption technologies (TripleDES, AES, IDEA), digital signatures (RSA, DSA, DH), cryptographic hash algorithms (MD5, SHA-1)
• host system security issues, from both a user and system administration perspective (backups, patches) [6]

It is crucial that one division of the team possess a thorough understanding of the full range of technologies and issues used by the team. This contributes to expand and intensify the technical resource and capability of the team and train other team members through education and documentation. It also makes sure that the team can provide a full range of services. Besides an in-depth understanding of the technical skills listed above, the following specialist skills are required:

• technical skills such as programming, administration of networking components (e.g. routers, switches) and computer systems (UNIX, Linux, Windows, etc)
• interpersonal skills such as human communication, experience in presenting at conferences or managing a group
• work organization skills

Obviously, a team will be unable to employ individuals who possess all the necessary interpersonal and technical skills. But there are opportunities to address such deficiency in those skills, such as training of staff to develop and retain such skills and support continuous progress.

Hiring CSIRT Staff

For any staff vacancy, the hiring process to select the most talented applicant is a complicated task. Even a candidate who appears on the surface to possess the right skill set might not be able to work within CSIRT setting. It is true when a crisis has been declared where the candidate may not be able to cope with the situation and inefficiently carry out their duties. Therefore, it is recommended to present the applicant to a hiring process, specifically designed to reveal the applicant strengths and weaknesses. Based upon the findings of the hiring process, the team will make up their mind to train the applicant in the specific skills that the candidate may require or decide not to employ the candidate. Compared to a regular hiring process, additional steps should be included in any CSIRT hiring process and they are:

• pre-interview document check
• pre-interview telephone screening
• interviews that cover topics from technical abilities to interpersonal skills
• candidate technical presentation
• reference checks, including criminal records

The complete hiring process should be devised to detect potential employees who possess appropriate interpersonal skills and technical skills. Such candidates can undergo further training to acquire more competence. Before calling the applicant for a personal interview, the pre-interview document check and telephone screening determines in the first instance whether the candidate is an ideal match for the selection process. At this stage, more information is gathered about the applicant's broad level of interest in computer security and other more specific details on items covered in his or her resume. The telephone screening will give a good impression of the candidate's oral communication skills. Before CSIRT staff begin to interview potential candidates, it's better to decide in advance what particular issues ranging from technical issues and ethical issues to social skills are most likely to be discussed during the interview process and select which existing staff are most suitable to talk about those issues with the candidate. Thus separate topic areas are covered by each of the various interviewers, saving any duplication of effort. Each interviewer will be in a position to review and consolidate feedback on the issues covered. Another strategy may be carried out where similar topics may be discussed by other team members involved in the interview process to agree on the candidate's faculty about a particular topic and identify any weaknesses. To ensure proper recruitment, the applicant should have the opportunity to meet up with CSIRT team members through a lunch meeting or at the candidate's technical presentation. A candidate, required to give a technical presentation, offers CSIRT an opportunity to measure other technical and interpersonal skills of the candidate. It also gives an idea how much common sense the candidate has and whether the applicant will be able to cope under stressful situations. Other qualities such as overall presentation skills, an eye for detail, technical accuracy and ability to answer questions on the fly are also taken into account. After an individual has been appointed, there is also an enormous task to make them adapt to CSIRT. The new staff will need to undergo training for some period of time to get used to the CSIRT working environment as well as specific policies and procedures for the team. Some new recruits may be given access to limited information until relevant certificates or clearances such as government or military clearances are obtained. Staff training is compulsory in order to make the new recruits acquire the necessary skill level to take on their new responsibilities. Secondly, training is necessary to expand existing staff skills for personal career growth and overall team progress. Staff training also helps overall CSIRT skill set updated with emerging technologies and intruder trends. When considering the overall training needs of the team, it is necessary to spot out the overall skills needed for each individual, as well as the common skill set required for the whole team. Obviously, new staff member should acquire immediate training in any deficient skills to perform effectively quickly. From a general viewpoint, the whole team should be assessed to determine any training that needs more attention to enlarge skill set exposure in the team. At the same time, this assessment focuses on an individual's skill set. Policies and procedures are a necessity and should be enforceable to support initial training of new team member and to guarantee ongoing training as policies and procedures get amended. Besides the interpersonal and technical skills discussed earlier, each team member should be trained in areas specific to the incident handling functions in a normal CSIRT work environment. Training should cover up the following issues:

• new technical developments
• CSIRT team policies and procedures
• incident analysis
• maintenance of incident records
• understanding and identifying intruder techniques
• work load distribution and organizational techniques

Initial training is conducted through on-the-job training. Since incident handling profession is different in work nature from other professions, there is no formal educational path for CSIRT staff and limited documentation in the literature. Most printed material come in the form of workshop reports or presentation slides. New recruits can examine and study internal materials, such as policies and procedures, case studies or past incident archives. Due to these limited documentation, on-the-job training is a must. CSIRT is aware of this gap and has built up a suite of training courses and other supporting materials to help new and existing staff. Such training resources are aimed at managers and technical personnel in areas such as creating and managing CSIRTs, responding to and investigating security incidents and enhancing network security. When handling sensitive information, even the most experienced CSIRT staff undergoes some level of stress. This results from the scale of consequences if they handle the information improperly. Initially, new staff can be flooded with extensive information, policies and procedures. As a general rule, it is not advisable to assign such staff to duties where they might disclose sensitive information without some initial training. It is more appropriate that the beginner learns the profession without making costly mistakes. To ensure smooth operation, existing staff should mentor new staff accordingly regarding the team's policies and procedures through on-the-job training. The new staff should first watch the proceedings of the experienced staff member and discuss any area of confusion. As they become more accustomed to CSIRT environment, the new staff might start to draft document for review and edit by the experienced team member. Consequently, the new recruit will gain competence in the areas of triage and request handling before tackling small-scale incidents. The new staff will be then able to handle tasks without assistance.

CSIRT Mission Statement

CSIRTs, which are already in existence, don't possess a proper knowledge of their aims and objectives or have even been unsuccessful to competently convey that information to the parties with whom they deal with. Consequently, they pointlessly use up effort and resources in an attempt to

• understand if they are using the correct priorities to ensure they respond to the most activity
• correct any inappropriate expectations of those they interact with
• understand how and whether it is appropriate for them to react to a given situation
• revise their policies and procedures to meet the needs of the situation
• determine of the range and nature of the services they offer should be modified [7]

A CSIRT must have top priority to identify, record, conform to a clear mission statement which needs to be circulated to concerned parties. Failure to do so, the situation is unlikely to progress further. Owing to its significance, the mission statement should be well-defined and be composed of about three or four sentences, indicating the task with which the CSIRT is accountable for. The team will have a simple insight of what achievement it is expected from them and most essentially, it will provide a focal point for the overall goals and objectives of the CSIRT. Moreover, senior management such as head of Information Technology, corporate security officer, board of directors must implement and support the mission statement of CSIRT. Without such support, the CSIRT will strive to gain admiration and adequate resources. The mission statement's purpose is to assist the team to initiate a service and quality framework. The latter includes three derivatives, namely the nature and range of services provided, the definition of its policies and procedures and the quality of service. Methods and their approach used to undertake the mission statement are the services offered by the team. Policies are leading benchmarks adopted by the team. Quality is the wanted outcome at which all activities will be undertaken. All CSIRT events are guided and controlled by the service and quality framework. In addition, many CSIRTs publish a purpose statement that supplements the mission and clarifies the motive behind the setting up of the team. Along with this purpose statement, CSIRT should be well equipped to define its goals and right services to reinforce its mission. These statements should be made public to other parties who will certainly work together with the CSIRT during the course of events. It will help out to give a clear understanding of CSIRT's role, purpose and the framework within which it operates.

Incident Severity Levels

Incidents are normally classified into different severity levels. The level of incident severity and their criteria depend on the needs and priorities of the company.

Security Incident Response Phases

Preparation Phase

The preparation phase looks at setting up a series of strategies, processes, procedures and agreements that will deal with the supervision and response to security incidents. It is typically comprises of guidelines identifying levels and responses, auditing and logging, reporting guidelines, resolution and follow-up.

Alert Phase

Gathering preliminary details about a potential security incident and reporting to the IT department of the organization take place in the alert place. The latter will instantly refer this to the Computer Security Incident Response Team (CSIRT). Diverse sources can generate alerts. These include monitoring of firewalls and intrusion detection systems, anti-virus software, threats received via e-mail, and media reports about new threats.

Triage Phase

The triage phase gathers available about the situation to deduce whether or not a security incident has occurred. In the event of an incident, the type of the incident is established, the initial priority level is allocated and the recording of all steps performed is activated. The incident component lead is assigned. In this context, an Incident Response Team (IRT) is also set up to execute actions related to incident handling. Both options, choice to pursue or protect, are available according to the sensitivity of the data and criticality of the operational system. If a choice to pursue is declared, it assumes the disruption or abuse continues as IT staff go on collecting data about the incident prior to proceeding to protect the system and launching steps to halt the expansion of the crisis as in the containment and eradication phases. Whichever options are chosen, defensive actions will be carried out on the system to safeguard data and system resources on the affected system. Higher priority level incidents require special attention which is generally attributed to potential legal or public relations impacts emerging from each course of action.

Response (Containment and Eradication) Phase

The response phase copes with restraining the extent and magnitude of the incident in order to keep it under control. Major concern is given to issues such as system backup, the threat of continuing operations and changing passwords or access control lists on the compromised systems and data. This phase also deals with finding out the root of the incident, clarifying system vulnerabilities, improving system defenses, removing the cause of the incident to eradicate any likelihood of recurrence. An alternative recommendation is to initiate Business Continuity plans.

Recovery Phase

During this phase, business operations return to its habitual operational state. Tasks consist of restoring and approving the system, monitoring the current state of system and business processes to validate normal operations without further system or data compromise.

Follow-up Phase

The follow-up phase deals with the issue of a comprehensive incident report which is circulated to concerned entities according to established policies. This phase also reveals lessons learned from the incident handling process, taking into account both effective and unproductive actions taken in response to the incident. It also builds up suggestions to prevent future incidents and improve enterprise security performance. In addition, many of these authors provide a set of processes or steps that are used in incident response activities. Selections of these processes from several authors are highlighted in Appendix B. Each process is outlined from each of the books or articles reviewed. In reviewing the materials in the appendix, it can be seen that the basic steps for incident management and response are very similar across the authors. They basically break down into some form of

• prepare/protect
• detect
• contain
• analyze
• respond
• improve

The “prepare” or “protect” functions refer to proactive mechanisms to have in place to effectively respond to an incident. This includes having incident reporting guidelines available to the constituency and defined incident handling procedures for CSIRT staff. It also involves the implementation of security best practices to protect systems. These best practices can include applying appropriate security configurations for software and hardware; keeping up to date with patches and operating system upgrades; monitoring system and network activity; disabling unneeded services; enabling maximum auditing; installing internal and external defenses such as firewalls, routers, and intrusion detection systems; and raising user awareness regarding computer security issues. In the CSIRT community, we often say “Reuse, with appropriate attribution, is good.” Being able to learn from the actions and experiences of other response teams can be very effective in helping a team to develop their own plans. For example, in building a team for the German Research Network back in 1993, Kossakowski pointed out that they were able to gain a lot of knowledge about what they needed to launch their CSIRT from talking to other teams [Kossakowski 94b]. He also pointed out that it can be challenging to prepare a successful plan for a CSIRT, especially if starting from scratch. One of the lessons learned was that talking with other teams, reviewing information that is available on CSIRTs in general, and where possible and appropriate, visiting other CSIRTs, will go a long way towards helping you to build an effective plan for your own CSIRT [Kossakowski 94a].

Receiving Incident Data

CSIRTs are equipped with some traditional methods to receive reports, alerts or requests. Reports can be emailed to a standard email address for CSIRT. Other points of contacts are CSIRT hotline or helpdesk, online or paper-based form that can be filled out to report an incident. Automatic alerts and reports are generated by IDS (Intrusion Detection System), network monitoring programs or other network sensors.

Recording and Tracking CSIRT Data

Newly created CSIRTs may not distinguish what they need to collect and the way in which tracking and recording of information need to be accessed, used and archived. Thus, tracking and recording of information in a consistent and systematic approach pose a challenge for them. According to (Practical UNIX & Internet Security, Garfinkel and Spafford), two rules need to be followed when responding to incidents.

• Rule 1: Don't Panic!
• Rule 2: Document!

Tracking and recording of information is necessary for the following reasons:

• preserving an archive of the kinds of incidents that were handled over the course of the year
• classifying and collecting trends and statistics
• safeguarding information as evidence that will be admissible in a court of law
• distributing incident workload across staff
• generating statements on incident status or incident report
• categorizing work tasks for completion of an incident
• assigning an incident to another staff for completion
• pinpointing large-scale incidents that might not be obvious when examining individual incident reports
• comparing incident activity across the company

CSIRTs should ensure what data should be recorded and tracked. This will promote effective response services for the team and ensure that the team is complying with any management and funding obligations. This also helps to clarify whether more staff is needed. CSIRTs propose the introduction of a log to start securing critical information about a reported activity. Details can be stored in a sequential manner, such as the date and time of the incident, what has happened, who has been contacted, what steps have been taken or need to be taken. These logs are in demand during periods of time when incident activities are on the rise and CSIRT are handling multiple events. This compilation of allocated incidents may include the following items:

• recently received reports
• constant interactions with previously reported incidents
• communicating or working together with other CSIRTs which may assist in dealing with similar incidents
• reopening of closed incidents since additional activity reports were received.
• other types of requests for information

Making an effort to remember specific details for a particular incident poses a challenge, especially when hundreds or thousands of hosts are implicated in an incident, remembering what happened, who was the point of contact. In the event of recording and tracking CSIRT data, it is wise to design that method in a proper way so that all data can be easily revealed and the responsibility for handling incident reports can be reassigned to other staff members. Existing summaries of the workload and distribution across CSIRT staff can be established. Another factor to consider when selecting the type of system for recording and tracking information is to determine whether it contains integrated processes for collecting and storing data from other sources such as telephone calls, fax and other types of communication.

Data Fields

A standard incident reporting form, developed by CSIRTs, collects details that will help to classify the “who, what, when, where and how” of the reported incident activity. For example, a typical incident reporting form contains a standard set of data fields used to gather critical incident information:

• contact information of the individual reporting such as name, phone number, email address
• organization reporting incident
• organization where the incident took place
• time of incident
• time of report
• time of incident discovery
• summary and technical details of the problem encountered
• type of systems affected
• operating system of those involved hosts
• IP address(es) of those affected hosts (source and destination of attacks)
• incident resolution steps and actions taken Apart from the above data fields, check boxes are included in the reporting form to identify
• the detailed type of incident activity that has occurred
• whether the incident is still continuing or has been ceased
• key points to measure the extent of the damage and threat severity level CSIRTs should

Teams also need to identify how long they retain information about their incident reports. Some teams will keep information for short periods of time (months), while others may keep information for several years.94 Different types of teams for different sectors may have various legal requirements that impact how long they can retain information. Although currently there are no widely accepted standards used by the CSIRT community to record and track CSIRT data, there is ongoing work in the IETF to develop standard data formats for exchanging incident data between teams. For example, the IODEF defines a common data format for describing and exchanging incident information [Arvidsson 01]. IODEF has been designed to be compatible with the Intrusion Detection Message Exchange Format (IDMEF) developed for sharing intrusion detection data between intrusion detection systems [Curry 03]. More about this project can be found in Section 3.10.2.1 of this report.

3.7.5.2 Mechanisms for Recording and Tracking Incident Information

Information can be captured and logged in a variety of ways: on paper or in a logbook, in a database or help desk system, or even in text files. When participants were queried in the CSIRT Organizational Survey about how information was collected, they responded as follows:

• 76% of the teams stated that they used a database to record and track incident data
• 28% use both a database and a paper log
• 10% use only a paper log
• 45% said they used a customized database
• 28% said that they used an off-the-shelf product

There was no particular database product used consistently by the CSIRTs. Products mentioned included

• Remedy HelpDesk and Action Request System
• SQL
• Oracle
• Microsoft Access
• Lotus Notes

A number of teams find they must build customized environments to collect, record, and store CSIRT information because some of the tools do not have the features needed or do not meet the functionality required by the CSIRT. One example of some of the work that is currently being done to create a customized incident handling tracking system is the development of Request Tracker for Incident Response (RTIR). 95 JANET-CERT is currently funding a project that has led to the development of RTIR, which is a customized version of an earlier general-purpose tracking system called Request Tracker.96 JANET-CERT worked in collaboration with the software vendor, who did the actual programming. DFN-CERT is also working on a project that has been funded by the German federal CSIRT, CERT-BUND, that involves researching requirements for a CSIRT incident tracking system. The prototype developed by DFN-CERT is based on RT and its extension for incident response, RTIR. The prototype is web-based and is called “Vorfallsbearbeitungssystem,” or VBS for short. The VBS extends RT/RTIR by adding specific workflows via roles and data fields specific to incident handling.97 Staff from both projects are collaborating and continue to work with the software vendor to extend the requirements for an incident response specific tracking system. They are also looking for other teams who are interested in developing other extensions to this software.98 Some other examples of incident tracking systems that are being developed and used for incident tracking within the CSIRT community include

• the CERIAS Incident Response Database, which has specific fields for capturing incident costs
• the University of Chicago Network Security Center (NSC) Freeman Incident Tracking System (FITS)

3.7.6 Categorizing and Prioritizing Incident Reports

There is no clear consensus on the best way to categorize and prioritize incident reports and activity. A variety of ways to identify and prioritize reports have been used by different organizations and discussed by various authors. A few of these are summarized in Table 12 Most authors believe that something as simple as establishing or assigning rankings such as Category 1, 2, or 3 or High, Medium, or Low will assist in prioritizing incident reports. Over time these may need to be expanded to meet the requirements or needs of the CSIRT and constituency being served. At one of our CSIRT development courses, one attendee discussed their categories for intruder activity and response. They used colors as indicators for the level of “threat” associated with an incident or other activity being handled by the team: red (high priority), yellow (cautionary alert, has potential to escalate), green (everything normal).102 These alert banners would be strategically located in the CSIRT offices as visual reminders of current activity levels. The yellow alert was also used to let part-time members of the CSIRT know that they may get called in if the priority went higher. Because there are not consistent severity scales across CSIRTs, one of the more unfortunate problems that can occur is that scales can be contradictory. In some cases the priority scales used have just the exact opposite level of severity compared to some of the others. While a selected priority setting works very well within an organizational constituency, it could lead to confusion in those cases where incidents affect multiple sites beyond a single constituency base. If a clear understanding of the relative priorities or criticality is not understood by all, the response actions taken may seriously (and detrimentally) affect the ultimate resolution of the activity. In looking at the lifetime of an incident, it must be recognized that the priority of the incident may change as new information comes to light. The priority of a specific type of incident might also change over time as changes in mission and services occur.

3.7.7 Incident Response Processes

CSIRT response strategies vary as much as CSIRTs themselves do. The response that a CSIRT provides is based on its mission, services, and service levels. Response options can include

• providing guidance and solutions via phone or email
• going to the site or affected machine and helping repair and recover the systems
• analysis of logs, files, or other artifacts
• assistance in legal investigations and prosecution
• capturing and documenting evidence from affected computers
• development and dissemination of patches, fixes, workarounds, advisories, alerts, or

technical documentation

• notification to sites involved in the activity (both victim and source sites)
• none (forward to others to handle)

Once an incident report has been received and reviewed, the response provided will depend on the CSIRT's mission, purpose, expertise, and policies and procedures. For example, a state law enforcement CSIRT's mission may be to pursue legal investigations; when they receive a report, they begin an investigation to collect evidence for prosecution, and they are not responsible for helping to repair the affected systems. However, an internal CSIRT in a commercial company or an MSSP incident response provider, after analyzing an incident report, may go to the site of the affected systems and physically perform the recovery operations to collect forensic evidence and also repair the affected systems. In the CSIRT Organizational Survey, we were interested in seeing the level of involvement that CSIRTs had in the recovery and repair operations. We asked not only how CSIRTs affected their response but also who in the organization actually performed the repair and recovery operations. The majority of the CSIRTs reported that the type of response they provide is either advice via phone and email (74%) or the development and distribution of technical documents and alerts (59%). Only 41% say they actually perform the recovery and repair of affected systems. And only 21% pass reports on to others to handle. Trends by sector include

• All of the banking and finance, information and communication, other commercial, and 83% of the education CSIRTs stated that they provide guidance via phone and email as their primary method of response.
• None of the CSIRTs in the banking and finance, education, or information and communication sectors stated that they passed on incident reports.
• 75% of the non-profit CSIRTs said that they provide response via phone and email guidance or by passing on the incident. No non-profit stated that they repaired or recovered the affected systems.

Trends by CSIRT model include

• None of the ad hoc teams stated that they passed on incidents.
• Coordination centers: 100% said that they provide advice via phone and email and by publishing advisory. None stated that they passed on incidents.
• Combined teams: 100% stated that they provide response via phone and email guidance, while none passed on incidents.
• Distributed dedicated teams: 100% provide response by phone and email guidance; 66%also repair the systems themselves.
• Distributed part-time CSIRTs: 75% said they provide response via phone and email and

by repairing and recovering systems themselves. It makes sense that distributed teams would be involved in the actual recovery and repair of systems, as they are most likely located on-site, in comparison to centralized or coordinating teams who are not on-site and who provide more guidance and support functions. In the same way, the combined and coordination centers seem to rely more on coordination of response and mitigation strategies. The centralized teams had no particular set of response options across the participating teams; they provide response across all categories of response options.

3.7.7.1 Who Rebuilds Systems?

When asked explicitly who rebuilds and recovers any affected systems, the participating CSIRTs provided the following information:

• 59% of participants stated that the IT department, not the CSIRT, recovers and rebuilds affected systems
• All of the CSIRTs in the commercial sector said that the IT department recovers and rebuilds systems.
• Most of the other sectors stated that both the IT and CSIRT did recovery and rebuilding.

All of the teams identified as distributed dedicated teams said that only the IT department recovered and rebuilt systems. All other types had either IT or CSIRT or both. No matter where the CSIRT reported—to the IT department or security department, or if the CSIRT was its own department—there was no consistent answer to who recovered and repaired the systems; it was IT, CSIRT, or both.

3.7.13 Secure Communications Mechanisms Used

CSIRTs must protect confidential and sensitive data at all times. This often means that a secure method of communication must be established when this data needs to be collected from another source or shared with other appropriate entities. There are a variety of secure communications mechanisms used by CSIRTs. These can include

• public key cryptography
• secure faxes and phones
• secure intranets or extranets

The majority of the teams participating in the survey stated that they used Pretty Good Privacy (PGP) for secure communications (75%). This could be influenced by the fact that many of the participating teams were FIRST members and FIRST requires use of PGP. Other reported information included the following:

• All of the participating banking and finance teams use PGP and digital certificates.
• All of the education, information and communication, and other commercial CSIRTs use

PGP

• 63% of the military CSIRTs use secure intranets or extranets.
• 66% of the participating non-profit CSIRTs use PGP.
• All of the combined teams use PGP.
• 80% of the centralized dedicated teams use PGP.
• 75% of the distributed part-time teams use PGP and digital certificates to communicate securely.
• The distributed dedicated teams use the most number of tools, with 66% using PGP, digital certificates, secure phones, and secure intranets and extranets.

3.3 Triage Function

The goal of this function is to ensure that all information destined for the incident handling service is channeled through a single focal point regardless of the method by which it arrives (e.g., by email, fax, telephone, or postal service) for appropriate redistribution and handling within the service. This goal is commonly achieved by advertising the triage function as the single point of contact for the whole incident handling service. If a team wants to limit the ability of constituents and others to bypass the triage function, direct contact information for individual team members (such as telephone numbers or email addresses) should never be given out. Because this is a common requirement across many CSIRT services, teams usually advertise a single point of contact for the whole CSIRT; and, regardless of the service required, a single triage function is provided for all the services that the CSIRT offers. Example: Within DFN-CERT, the person undertaking the triage function is called the CERT Hotliner. This person is responsible for reading all email to the response team's alias, opening all postal mail, reviewing incoming faxes, and answering all telephone calls. The DFN-CERT hotline and the personal telephone lines for all other team members are forwarded to this person's telephone to ensure that all incident-related calls are dealt with centrally. To stimulate the reporting and the collection of all relevant information, the constituency must be provided with easy to use and efficient mechanisms for reporting:

• a clearly defined point of contact
• specific details on the availability of the defined point of contact
• simple but defined procedures to follow
• clear guidelines on the kind of events to report
• supporting documentation (e.g., reporting forms and references to other available documentation) for reporter use

Once the information is received by triage, an acknowledgment of receipt will be sent, then the information will be sorted, prioritized, tracked, and passed on to other functions within the service. Additionally the triage function must decrypt encrypted messages and check digital signatures, preserve this information for later use, and allow for actually reading the content. To undertake this task, it is necessary for the triage function to have access to the data repository used by each of the other functions of the incident handling service. Based on the information content and the data in the repository regarding existing service events, an initial sorting will take place to identify which function of the incident handling service should handle the information. The next step is to determine if the information is directly related to any current or past event. If it is directly related to some existing or previously tracked event, it will be tagged as part of that event. Otherwise it will be tracked as a new event of a given type and tagged appropriately. In addition to being sorted and tagged, the triage function commonly assigns an initial priority to the information in accordance with the priority scheme in use by the functions within the service. If information enters in the form of hardcopy materials, it is common for the triage function to ensure that this information is entered online or a reference made online to the physical location of the materials. Tools for entering, accessing, and tracking information and events can greatly facilitate and semi-automate data manipulation and searches. Such tools can support the staff responsible for triage by helping establish the identification of

• new events (incidents, requests, vulnerability reports, other information notices)
• information directly related to currently tracked events
• information directly related to a previously closed event
• events that are being tracked separately, but may have a direct relationship
• information that is considered out of the scope of the incident handling service

If the information contains insufficient detail or is incomplete, it is likely that the triage function will become slow, inaccurate, or incapable of serving its role. In such cases it may be necessary to seek more detailed information from the sender before the information can be appropriately triaged, which delays the process. In addition to direct tool support for the triage function, other steps can be taken to enhance the quality of the information, such as tracking numbers, standard reporting forms, and preregistration of contacts. The next three sections deal with these topics.

3.3.1 Use of Tracking Numbers

If a team uses a tracking number scheme and can encourage or require others to use the assigned numbers in all follow-up correspondence, this will greatly facilitate the triage process. To facilitate automated support, the numbering scheme should provide simple identifiers for human and tool recognition. In a robust tracking system, the tracking numbers