Intelligent Vehicle Using Multi Sensor Data Fusion

Published Date: 02 Nov 2017

Alone sensor is capable of detection of obstacle but that decision may be false or incorrect. Laser scanner some time provide false data if laser hits to road pitch. Stereo vision camera is also capable of finding the obstacle by using V-disparity algorithm, but it cannot find the distance between the vehicle and obstacle. One sensor is not capable of taking a decision, so that multiple sensors are used to take decision. This decision is more accurate and precise. Proposed system is cooperative fusion between Laser Scanner and Stereo vision camera in order to minimize the drawbacks of sensors by the other sensors and generate the result robust, accurate and real time. In this system Laser scanner is used to detect the different obstacles with their distance between vehicle and obstacle. Stereo vision camera is used to give the confirmation of multi obstacles. Proposed system is used for the driving assistance purpose. This System have minimum false ratio so that this can be used in the real life scenario.

Keywords: Multi sensor data fusion, Laser Scanner, Stereo Vision Camera, V-Disparity algorithm, Intelligent Vehicle, Obstacle Detection.

Introduction

Perceptive systems are very powerful and widely used system in ADAS (Advanced Driving Assistance System). This type of systems are widely used for various sub-systems particularly obstacle detection, lane change assistance etc. in obstacle detection is necessary for pre-crash, collision mitigation, stop & go, obstacle avoidance or inter distance management. Only one sensor is capable to detect the obstacle but also having high false rate. To avoid the errors we can use multiple sensors to detect the obstacle accurately and very low false rate. Currently many vehicles uses ADAS systems for examples, reverse car assistance system, obstacle detection etc. mostly high values vehicle manufacture uses this type of systems for driving assistance purpose.

In the context of Intelligent Vehicles and Driving Safety Assistance, on board road obstacles detection is an essential task. It must be performed in real time, robustly and accurately, without any false alarm and with a low detection failure rate. Firstly, obstacles must be localized on the road, if possible with the best accuracy; additional information such as height, width and depth can be interesting in order to classify obstacles (pedestrian, car, truck, motorbike, etc.) and predict their dynamic evolution.

Basic concept of proposed system is to use multisensory devices to detect the obstacles present on the road and according to the distance between the vehicle and obstacle it gives the warning to the driver. In this system we use Laser Scanner and Stereo Vision Camera as sensors. Laser Scanner is used for the long range obstacle detection and stereo vision camera is used for the confirmation of obstacles. In particular, using stereo vision and a laser scanner seems to be an efficient solution. In order to provide better performances with such a configuration, we propose in this paper an innovative fusion scheme. In this approach, the obstacle detection and tracking tasks are performed thanks to the laser scanner. The stereo vision is used subsequently to confirm the detections. Perception range enhancement techniques are included in the confirmation task.

Literature Review

Obstacle detection us a powerful activity for autonomous systems in particular with ITS (Intelligent Transport Systems), Vehicles can be considered as a robots. The development of ADAS such as pre-crash, collision avoidance, collision migration or automatic cruise control requires that reliable road obstacle detection systems are available. There are a various approaches for detection of obstacles depending on the different types of sensors.

Obstacle can be detected by stereo vision camera; this concept was kept by Franke and Joos in2000. Depending on stereo vision camera some algorithms are developed so that they can provide robustly and with real time performance. The surface of the road including pitch, roll and non-flat geometry and then extract objects with respect to the estimated surface height and width of the objects can be evaluated. Road can be detected using a vision based lane detector and objects which are out of the road can be removed. However Laser Scanner data is used for the finding out the position and width of the obstacle.

So far, client/server computing model has been most popularly used in Distributed Sensor Networks (DSNs) to handle multi sensor data fusion. However, as advances in sensor technology and computer networking allow the deployment of large amount of smaller and cheaper sensors, huge volumes of data need to be processed in real-time. Multi-sensor data fusion technology concerns the problem of how to fuse data recorded from multiple-sensors, together with knowledge, in order to make a more accurate estimation of the environment and allow for a variety of applications, such as intelligent transport systems, traffic control, maintenance engineering, remote sensing, robotics, environment monitoring, global awareness, and others. Multi sensor integration can include data from various types of sensors that measure different environment variables, sensors of the same type that measure the same variable, or a combination of both scenarios. The basic problem is to determine the best procedure for combining input data from multiple sensors.

With the increase in number and type of sensors available, and the need to manage a growing quantity of information produced by those sensors, emerged the need to fuse those data into high level information that a human can perceive and could act automatically in the environment. This led to the need for multi-sensor data fusion technology. Data fusion first appeared in the literature in the 1960s, as mathematical models for data manipulation. It was implemented in the US in the 1970s in the fields of robotics and defense. In 1986 the US Department of Defense established the Data Fusion Sub-Panel of the Joint Directors of Laboratories (JDL) to address some of the main issues in data fusion and chart the new field in an effort to unify the terminology and procedures. The present applications of data fusion span a wide range of areas: maintenance engineering, robotics, pattern recognition and radar tracking, mine detection and other military applications, remote sensing, traffic control, aerospace systems, law enforcement, medicine, finance, metrology, and geo-science.

In a typical scenario, a combination of vehicles in different position of the road are facing various driving conditions such as driving straight, turning, overtaking a vehicle, meeting pedestrians, etc. For these different conditions, the ADAS equipped vehicle need different sets of sensors to detect environment-related information and determine a correct driving condition. The decision on selecting a proper set of object-detecting sensors should be made based on the capability of available sensors and real-time driving condition. Now, In order to formulate and simulate the selection of object-detecting sensors with respect to various driving situations, the sensors should be capable of evaluating Driver commands (steer angle setting, backing, changing lane, turning a corner and overtaking a vehicle), Relative Vehicle's Velocity, Traffic Flow (Low or Dense), and Driver's behavior (Observant, sleepy, drowsy, aggressive, using cell phone, etc.).

The combination of these parameters will be used to reflect a proper diving situation encountered by the driver. So we need an optimal selection of some appropriate sensors to monitor all these four factors. But, which sensor is better and optimal? Image sensors have some drawbacks, such as low ability of sensing depth and advantage of higher ability of discrimination than LIDAR and RADAR. Radar shows limited lateral spatial information because it is not available at all, the field of view is narrow, or the resolution is reduced at large distances. Although LIDAR has a wide view field that solves part of the previous problems, there are other problems such as low ability of discrimination, clustering error, and recognition latency. These restrictions of the different sensor types explain the attention given to sensor fusion in research on object detection and tracking (Cheng et al., 2007). According to advantages and drawbacks of mentioned sensors and various real driving situations

Instead of Stereo vision camera we can use CCD camera and CMOS camera, but both the alternative having disadvantages, these disadvantages is overcame by using stereo vision camera. CCD camera technology limits the frequency of the detection to the video frame rate (25 Hz) detection failures can be observed for blooming effects, due to sun or vehicle lights in the night conditions. On the other side CMOS camera produce noisy data. This may cause error in the finding obstacle due to this false alarm maybe raised when the road is non-constant roll.

Stereo vision camera can detect the obstacle up to few meters, because of the limited detection range. Usually fusion algorithms inn the road context use homogenous data concerning obstacles position obtained from Laser scanner and hardly take into account complementary features to enhance the detection results.

System Architecture

Proposed system is a multisensory data fusion approach between the Laser scanner and stereo vision camera in order to take advantage of the best features and cope with the drawbacks of sensors to perform robust, accurate and real time. Obstacle detection in the vehicle to be exploited for driving assistance purpose, a system must have low false detection rate with this idea in mind we propose a system with Laser scanner and stereo vision camera for road obstacle detection. The obstacles are tracked by the Laser scanner and then stereo vision camera is used for the confirmation of obstacles on the road.

Fig. 1 Architecture of proposed system

Laser scanner and stereo vision camera are complementary sensors, after fusion the final position of the detected obstacles is provided by Laser scanner, which is more precise and accurate than the stereo vision camera. Laser scanner provides the width and depth of the obstacles whereas height of the obstacles as well as road geometry is given by the stereo vision camera. The road geometry can also be provided by stereovision and it could be used to avoid some false alarms from laser scanner. The question is then to know how the data provided by stereovision and laser scanner can be combined and/or fused together in order to obtain the best results.

Laser scanner scans the road for detection of the obstacle. If any obstacle is detected then the raw data is provided to the clustering algorithm for clustering of multiple obstacles. After clustering of object given by the raw data, it will give x and y coordinates, height and width of obstacle. If obstacle is detected at the same time stereo vision camera will gives the geometry of the road and height of obstacles. Data from clustering object whose having x, y coordinates, width and depth of the obstacle is filtered using image taken by the stereo vision camera. Same time laser scanner gives the false alarms then using stereo vision camera we will avoid that false alarms. If any obstacle is detected by the laser scanner then it will confirm by the stereo vision camera. Vehicle having different sensors like speedometer, accelerator any many more it will provide the data to the threat assessment section, and to detect the obstacle is harmful or harmless. It will also give the raw data to the warning strategies section. At the warning strategies we will give the warning to the driver about obstacle. It will do by the two ways whether driver assistance system give warning visually and by audio signals, or automatic actuators takes action against obstacle.

Laser Scanner Raw Data Filtering and Clustering

The idea is here to use the geometric description of the road provided by stereo vision in order to filter the laser raw data or clustered objects which could be the result of the collision of the laser plane with the road surface. Two possibilities are available:

1. Firstly, remove impacts that could be the result of the collision of the laser plane with the road surface from the laser raw data; secondly, cluster impacts from the filtered raw data,

2. Firstly, cluster impacts from the laser raw data; secondly, remove clustered objects that collide partially or totally with the road surface.

Removing Objects Out of the Road

Once objects have been detected from laser scanner, they can be filtered with respect to the road lane position. The same filtering can be performed for objects detected by stereo vision.

Simple Redundant Fusion

At this step, filtered objects from laser scanner and stereo vision are available. The idea of the first fusion strategy is very simple. It consists in introducing redundancy by matching the set of obstacles detected by stereo vision with the set of obstacles detected by laser scanner. If an obstacle detected by laser scanner is located at the same position than an obstacle detected by stereo vision, the obstacle is supposed to be real; otherwise it is removed from the set of obstacles.

Fusion with Tracking and/or Association

More complex strategies consist in introducing feedback, by using objects tracking and/or association. The idea consists in performing multi-obstacles tracking and association for each sensor in order to obtain multi-tracks for each sensor, and then to perform multi-track association between the tracks of the stereo vision and the laser scanner, and eventually to fuse the tracks in order to increase the certainty of the tracks. Figure shows present a fusion scheme including tracking and association for both stereo vision and laser scanner sensor, and global fusion.

Threat Assessment

On the basis of above we get the length, breadth, height and width of the obstacle present on the road. Input of this section is all the coordinate of the obstacle and input of the other sensors, depending on the other sensors like odometer, steering angle we will assess the threat, whether the threat is harmful or harmless. If the threat is harmful then how much impact on the vehicle, that will be calculated by this threat assessment section

Warning Strategies

Warning strategies of two type whether it will be HCI (Human Computer Interaction) or Actuators. In the HCI we can give warning by visual display or by haptic technology or by audio signals. And in the actuators we can actually perform some action on the incoming threat. If the any vehicle is standing on lane, then it will detect the distance between two vehicle, then calculate the impact on our vehicle, if the impact is too high then it will take actions, whether by HCI giving warning to the drivers and give suggestions by visually or by audio signals. In actuators it will automatically take actions for examples lane change, automatic brakes etc. warning strategies is very much important in the intelligent vehicle to warn the driver about incoming threat.

The "v-disparity" Algorithm for Stereo Vision Camera

V-disparity algorithm is very important for obstacle detection. This is developed for to avoid the computational burden of the disparity images. The v-disparity algorithm can be considered as a 3D graphical representation of the similarity measures between left and right images. First, a pair of stereoscopic images is grabbed. A sparse disparity map is then computed. The "v-disparity" image is built and global surfaces are extracted. The position of obstacles on the road surface is then deduced. The road surface must be extracted before the obstacles because obstacles are defined as objects located above the road surface.

Figure 2: The "v-disparity" Algorithm

K-Nearest Neighbor algorithm for Laser Scanner (KNN)

KNN is a simple algorithm to clustering various data based on the similar measures. KNN has been used in statically estimation and pattern recognition. Basic aim of KNN algorithm in the proposed system is to clustering the objects and depending on the objects it will classify different objects and gives the information about clustered objects like x and y coordinates width of object and depth of object.

KNN assumes that the data is in a feature space. More exactly, the data points are in a metric space. The data can be scalars or possibly even multidimensional vectors. Since the points are in feature space, they have a notion of distance – This need not necessarily be Euclidean distance although it is the one commonly used. Each of the training data consists of a set of vectors and class label associated with each vector. In the simplest case, it will be either + or – (for positive or negative classes). But KNN, can work equally well with arbitrary number of classes. We are also given a single number "k". This number decides how many neighbors (where neighbors are defined based on the distance metric) influence the classification. This is usually an odd number if the number of classes is 2. If k=1, then the algorithm is simply called the nearest neighbor algorithm.

Fig. 3. Example of a result of clustering