The Network Monitoring Services

Published Date: 02 Nov 2017

Abstract

Communication networks have become an indispensable part of our life today. A failure in a large and complex network such as Lancaster University campus network ,is difficult to detect and diagnose .It is this problem of network failure ,which we are trying to resolve in this report .Network measurements in such scenarios provide us very valuable information about the performance ,utilization ,QOS and health of network .However complexity and seize of the network makes these measurements a daunting task .Lack of administrative tools to access backhaul network ,multiple points of failure and a need to make these measurements in a to non-intrusive manner makes the problem much more challenging.

In this report we present case study of Lancaster University’s campus network including the logical structure of the network, the composition and direction of traffic flow within network and services provided by Network Management centre to the users within the network .We then present the design of measurement tool which we are using to detect and diagnose different kinds of failure that occur in a network

Introduction

Over the years we have seen tremendous growth in communication network complexity .This complexity is also reflected in campus network, such as Lancaster University’s campus network .Lancaster university’s campus network is a hierarchical network comprised with 350 km of fibre optic network in use across the campus 1,500Km of copper lines cowering whole campus ,its having a capacity of 31,000 data points out of which 10,000 data points in use ,in addition to this 450 wireless network access points daily serving 5050 university work stations across the campus .

At the bottom most level we have a user ,accessing the network from his machine .A group of users in a building form a subnet ,which are connected to backbone routers through switches and hubs ,and finally there is NAT box cum proxy server which provides WAN access to users .Users has multiple applications running on his machines .Routers and switches perform the function of routing and traffic measurements and firewall tries to shape the network traffic .This growth in a complexity and distributed nature of the network and the network elements has made network measurements and monitoring a daunting task not only for administrators but for research as well.

It is simply not enough to monitor the status of just one element in the network ,there are multiple elements in the network ,which successfully interact with each other t provide end to end connectivity to users .thus complicating the task of network measurements and monitoring leading us to question of WHAT,HOW,WHEN and WHERE to measure .In such large and complex networks like Lancaster University network ,there are multiple points of failure like input buffer at intermediate router might be full and packets might be getting dropped or the proxy server might be overloaded and dropping requests or some WAN link might itself down or domain name is not resolved because of underperformance of DNS or their might be some miss configurations at the user’s machine and traffic from other users also effects the performance of the user .This ends with a question in every users mind "What went wrong with this network".

In this report we try to answer this question by performing measurements on the campus network to detect and diagnose failures in campus network .We also perform some measurement studies to determine the performance of Lancaster university network

1. Lancaster University Network Overview

The Lancaster University Network is complex and vast entity network, campus network is hierarchical network with 350km of fibre optic network in use across the campus 1,500Km of copper line cowering whole campus .it is having the capacity of 31,000 data points out of which 10,000 data points in use in addition to this 450 wireless network access points, daily serving 5050 university work stations across the campus, in total 900 networking devices in 168 communications room providing 6,625 residents with high speed broadband.

Daily service to 5050 university workstations across the campus providing network access to large group of faculty, students and staff .Round the clock monitoring and management of such a network is a daunting task to .To understand the challenges involved in measurement of such a network is important to first study its structure, understand the kind of traffic that flows through it and services that run over it

This section outlines a structure of network in Lancaster University and gives a description of its various components emphasizing on the problems faced by different group of people using the network

1.1 Network Topology

The campus core network is arranged in hierarchy, with three main routers located in different location in campus, main routers includes border gateway router situated near Graduate North Avenue, along with eleven other routers located in different parts of the campus controlling the traffic flow. Campus network can be classified by considering different type of users that is

Hostels: Access to students and the staffs

Academic: Access to the academic areas

Admin

Wireless :Wireless access to selected location in the institute

Physically wire from an office ,lab and user machine goes to one of several communication equipment rooms in the building and connects into hub or switch ,these hubs are interconnected via either a shared Ethernet segment or switch in the building’s main communication equipment room .This hub or switch is then connected via fibre –optic cable to router port in one of the campus router ,using switches in buildings ,network traffic in different building is kept separate ,that is computer in one particular building cannot see the traffic from any other building

Lancaster University local network is connected to external network via leased broadband lines (32 Mbps, 8Mbps, 64 Mbps).traffic destined to nodes outside campus is filtered at the router kept near Graduate North Avenue building.

2. Network Stakeholders and Application

Network stake holders of university network can be grouped into two different categories based on the network usage and the network monitoring

University network operator: provides ,manages and monitors the various network services to all the users in the institute .These services include electronic mail, FTP, World Wide Web , DNS and many other Services

Users (Students, Staff, etc) :This category consist of major stake holder o f university network. This is categorise by considering different type of users who are availing different type services from university network, its mainly consists of students who is mainly contributing to the network traffic and network usage of. Teaching staffs are the another kind of network users

2.1 University network operator:

University Network operate Manage centre manages and monitors various network services to all the users in the institute .The service includes electronic mail, FTP, World Wide Web, DNS and many other services ,Let us consider some of these services

1 Web Proxy

Squid proxy server serves the campus network as web proxy. It performs the function of web caching, content filtering, user authentication, ad-blocking and bandwidth shaping. As mentioned in Section, campus network is connected to Internet via three WAN links. Proxy server also performs load balancing on these WAN. Proxy server is essentially a cluster of machines which appears as a single server to end-users because of Ultra Monkey. This is essentially done by providing virtual server as front end and using real servers as back end

2 Firewall

A packet coming into or going out of the campus network has to pass through different Firewalls. The purpose of firewalls is to protect internal network in-case servers are compromised. These firewalls are implemented using, iptables the open source firewall tool. It consists of rules for how to deal with packets. These rules are grouped into chains, an ordered list of rules. Further, these chains are grouped to form tables, there are three basic tables containing some predefined set of chains. These are the Filter table, NAT table and Mangle table.

a)Filter table: is used to packet filtering. It is used to restrict the services available to the network users within campus. It does so by blocking the traffic on specific ports, for example outgoing packet has been blocked from everywhere except the School of Computing And Communication machines (which can be identified from their IP address). It also looks into the type of request to block certain services.

b)NAT table: is used for rewriting packet addresses and ports. It is through the use of NAT table that firewall also acts as the NATing agent. Connection tracking is done to keep track of states and expectations. SNAT is used for changing the source address while DNAT is used for changing the destination address. Requests forwarded outside the campus network contain the address of the three WAN interfaces provided by the ISP.

c)Mangle table: is used for modifying packet options and hence enables traffic shaping.TOS, TTL and MARK field of the IP header are modified. By changing the MARK field and using iproute2, specific routing is achieved.

3 Domain name service:

There is a local hierarchy of DNS servers within Lancaster University network. Some of the Building have a local DNS at lower level with Lancaster University DNS at the upper level. For example, INFO lab (10.129.1.1) and School Of Management (10.105.1.7) have their Own DNS servers at lower level and there is the Lancaster University DNS (10.200.1.11) for the whole campus. Queries for addresses with lancs.ac.uk suffix are resolved by the DNS at the local subnet or by the campus DNS, dns. lancs.ac.uk, while all other queries are forwarded to the DNS server provided by the ISP. There is a small cache of queries maintained at each of these servers. dnscache, an open source recursive name resolver is used to implement the DNS.

2.1(a) Network Monitoring Services

Network administrators provide some of the statistics about network performance using some of the commonly available tools. These include:

• Mail logs are also provided giving a count of number of incoming mails, outgoing mails and their sizes along with number of mails queued up in the IMAP server.

• Proxy Servers usage statistics is also provided using MRTG.

These services measure the load on the network but do not provide us the current status. For example MRTG measures the traffic load on network links but it does not tells us about the status of the given router or switch. Or for proxy for that matter. And all these services are running individually. The tool which we have built checks the status of the backbone of the campus network, which includes:

• Subnet Switch

• Subnet routers and CC router

• Proxy server

• DNS servers

2.2 Network Users: Students, teaching staffs and faculty the university are the major network users. Type of usage/application of network is different for different users. Students Each of this user have certain common usage characteristics, for example all of them still use email services. However, they have certain characteristics peculiar to their group. Academic users have higher web usage, accessing journal sites and HTTP content (text+ images) access. While hostel users largely access multimedia content and heavy traffic generating applications such as instant messengers. It is also known that a large amount of multimedia content is downloaded using HTTP ; this also contributes to the application traffic of the users.

3. Measurement Methodology

In this section we describe different parameters that are measured and method / tool used to measure network parameters in campus. Based on the type of measurements performed we classify these methods into, Active measurements and Passive Measurements.

3.1 Passive Measurements:

In order to not overload the network with measurement traffic passive measurements are performed. We snoop in on the packets coming in and going out from the host. Each node within a subnet can see the ARP packets and IP packets in that subnet. ARP packets however are blocked by the subnet router and are not allowed to go outside the originating subnet. IP packets on the other hand if destined for a host within subnet remain in the subnet or else are sent to the destination subnet via subnet router. Thus by looking at these packets on a host we can measure following parameters: Subnets reachable by the host Based on our assumption, that the clients connection settings are correct, receiving packet from a host indicates client was able to establish connection to that host and able to reach that host as shown in the We snoop in on the ARP packets to determine whether host is able to reach a hosting its own subnet. And snoop in on the IP packets (TCP and UDP) to determine which nodes are reachable outside its own subnet. Thus we record the reachable of the subnet switch, subnet router, DNS server configured at the client and proxy using the packets received at the client machine. If however in the given one minute measurement interval we do not receive any packet, we use it as an indicator for the subnet switch to be down. Since most of the time there are more than one user on the network, the subnet is always full of ARP packets thus if we don’t even receive any ARP packet in our measurement interval it can be safely assumed that subnet switch is not working. Figure

3.2 Active Measurements:

Passive measurement sometimes reveal information that at times is inconclusive, thus to make conclusive measurements active measurements are performed, however a bigger question with active measurements in such a large network is what to measure Lancaster University campus network is an hierarchical network, with fixed topology and static routing. The internal and internet bound traffic, , has a predefined fixed path in the network. Thus the task of active measurements is much more stream lined. Let us consider some of the network parameters that are measured using active measurements.

Packet Loss: loss over the link is one of the most studied properties of links. It can occur due to network congestion where the server or intermediate router starts dropping packets because of input buffer getting overflowed, it can also occur because of packets getting corrupted and being dropped at intermediate node or server.

Delay: Delay is measured as time taken by packet to reach from one machine to another in a network. It is a sum of propagation delay, queuing delay and transmission delay. Given the bandwidth and size of the link, propagation delay and transmission delay are deterministic. It is the non-deterministic nature of queuing delay which adds the randomness to packet delay and thus making its measurement harder. The lack of time synchronization across hosts in network makes the experimental determination of delay that much harder

Round Trip: Time or RTT is measured directly at the sender by measuring the time difference between the time when packet was sent and time when acknowledgment was received. There are several tools which give us RTT, such as Ping. It is throne way delay that is hard to measure. Time synchronization protocols are used to synchronize the clocks on two machines, which are then used to measure the one way delay.

Bandwidth: Bandwidth or throughput is the amount of data per unit time which is delivered over a link. In network measurements it is sometimes used exchangeable with link capacity, which is defined as the maximum amount of traffic which can be transmitted over the link. Thus bandwidth is data rate which we get from our experiments. Path char [15] measures the link bandwidth by sending variable size packets and performing a statistical analysis of the results to compute the link capacity. Path char is unique in its ability as it is able to measure bandwidth of all the links in path.

4.Types of failure which results in denial of network service

We tried to identify various failures that occur at various elements in network backbone. In this Section we describe the different types of application layer failures identified for Web Access, DNS servers and Routers.

Web Access Failure: Web access failure is the failure which is seen by an application when trying to access WAN from within the campus, following are different types of web access failures seen at the application layer failures.

503 Service Unavailable:

This error occurs after connection is established to the proxy ,however when it request proxy to establish a connection to the remote web server proxy returns this response .It indicates that proxy is up but it is running minimally as it is replying with the 503 Service Unavailable code. It can indicate either it is effectively closed for repair, or temporary overloaded such that starting new process would exceed the threshold of the limited connections.

• Connection timed out failure at Proxy: This error occurs when we try to connect to the Proxy. So either the proxy is down or it is so much overloaded that no new connection can be established. If multiple nodes in the campus observe this failure at the same time and none sees it up then proxy is down or else the proxy is overloaded.

• Connection time out failure at Server: This failure occurs when the connection times out because of no response from the server at the other end. If multiple users in the campus observe this failure for different URLs this indicates some problem with proxy, however if failure is observed for only one URL it indicates failure at the web server.

• Connection refused failure: : This error is generated by operating system call in response to the connect(), system call. It occurs when there is no service at the other end listening on the port number we are trying to connect to. This error occurs while connecting to nekton on port 80, indicating that the port 80 is not open on nekton at that time.