Analysis Basics Detection Of Driver Drowsiness

Published Date: 02 Nov 2017

The state drowsiness lies between the states wakefulness and sleep. It is still not defined when drowsiness starts and ends but the appearance is before Stage I sleep as currently specified by the American Academy of Sleep Medicine (AASM) [3].

The methods of detecting driver drowsiness can approximately be divided into 1) the driver’s behaviour, 2) the driver’s physiological signals and 3) the driving/vehicle behaviour caused by the driver. The driving/vehicle behaviour monitoring techniques analyse vehicle parameters such as vehicle speed, lateral position, steering-wheel angle and road design. These techniques do not disturb the driver but the outcome can be negatively influenced by driver types, vehicle design and the driving environment. Physiological responses like heart rate, muscle activity and EEG are intrusive due to the use of electrodes; whereas behavioural responses like eye and eyelid movements, yawning, nodding and gaze can be detected through non-intrusive image-analyse techniques. However, physiological techniques are the most accurate method to detect driver drowsiness and can also be used for evaluation purposes. The limitations of behavioural responses analysed through video processing are mainly the different illumination conditions and the ability to cope with restrictions such as obscuring facial features like eyelids, for instance, through glasses [4].

Indicators of Driver Drowsiness

A drowsy driver can be identified by noticeable problems in driver behaviour, physiological patterns and driving characteristics [1]. The following chapter gives an overview about basic effects of driver drowsiness.

The major indicators in terms of the driver’s behaviour to ascertain drowsiness are:

Increasing number of yawns

Changes in the positions of the hands on the steering wheel; drowsy drivers are more likely to have relaxed hand positions

More frequent touching or scratching of chin, face, head, ears, eyes and legs

Radical increase of eye-blinking activity

Decreasing head motion activity [1, 14]

A decreased eye closure speed, increased blink duration [15]

It is noticeable that in terms of eye-blinking activity the indicator PERCLOS (percentage of eye closure over the pupil over time) is a widely accepted measurement to identify drowsiness [12, 16]. It has been assessed by EEG data and subjective evaluation [1]. However, the most important indicators based on the driver’s behaviour to predict drowsiness, according to [17] and [15], are PERCLOS, average eye closure speed, amplitude / velocity ratio and blink duration.

Physiological patterns like EEG signals are reliable sources to estimate the driver’s drowsiness level. A downward trend of frequencies and the existence of lower frequencies are good indicators for sleepiness. Beta waves in the EEG signify an alert driver. Alpha waves, if appearing in the occipital regions, usually indicate awake and relaxed states. An appearance of alpha activity in the central or frontal areas of the brain is a possible sign of drowsiness, but is not required to define this state. With the increasing level of drowsiness, theta waves are presenter than alpha waves. Gamma waves indicate sleep [1].

A number of driving/vehicle behaviour characteristics can be used to identify drowsiness; here are some of the most important ones, which indicate drowsiness:

Fewer reversals of the steering-wheel [1, 7, 18]

Decrease of minimal corrections of the steering wheel [1, 7, 19]

Decreased steering velocity [1, 20]

Bigger amplitude of steering-wheel changes [1, 21]

Higher standard deviation of the steering-wheel angle [1, 21]

Changes of the steering wheel are more sporadic [1, 4, 7, 12, 22, 23]

Increased number of lane departures [1, 4, 7, 12, 22, 23]

Correlation between lane departures, lane deviation, standard deviation of lane position and eye closure [1, 4, 7, 12, 22, 23]

Parameters like mean square of lane deviation, mean square of high-pass lateral position and standard deviation of lane position have the possibility to become good indicators for drowsiness [1, 24]. Convincing results are also ascertained by looking at the yaw deviation variance and the mean yaw deviation. [20] Braking, accelerating and vehicle speed have not been found to be strong indicators for drowsiness [20, 25], although other studies show that the standard deviation of speed does increase with increasing drowsiness [26].

State of Art of Commercial and Research Systems for Detection of Driver Drowsiness based on image-analysis

The purpose of this chapter is to give an overview and an assessment about existing commercial and research systems to detect driver drowsiness. The focus lies thereby on detection systems which rely on video analysis of the driver’s face, because such systems appear to generate promising results (as discussed later). Current eye trackers are presented in the review as well, because they are the basis of an image-based detection system. Additional, low cost options are also introduced. The final goal is to highlight several image-based options, which are useful for the purpose of detecting driver drowsiness.

For further information here are a few other literature reviews: [27], [10], [1], [23], [4], [12]

Many driver-fatigue detection systems have emerged by various research groups, car- and automotive parts manufacturers. Error: Reference source not found in appendix Error: Reference source not found shows an overview of current systems. The objective of this table is to detect suitable systems and to collect information about, which main fatigue indicator and primary technology is promising and widely used.

First of all, a few relevant driver drowsiness detection solutions by car manufactures are discussed. The car manufacturer Mercedes-Benz revealed in 2009 its Attention Assist that collects data within the first twenty minutes of driving to generate an individual driver profile. This profile is compared with further driving behaviours and a classifier decides if ascertained changes are the result of drowsiness and if an alert is necessary, which would be an audio-visual signal. The main drowsiness indicators are steering wheel patterns along with about 70 additional factors like specific driver control actions. The Attention Assist utilizes only vehicle parameters and requires a certain developing time in the beginning to establish the drivers profile and is therefore not available during this period of time [1, 28, 29].

Volvo introduced its Driver Alert Control in 2007, which focuses mainly on the driving behaviour. A forward facing camera analyses road markings to determine, for instance, lane departure. Additionally, other indicators like steering wheel patterns and accelerator and brake pedal inputs are used to ascertain drowsiness. After a certain level of drowsiness is determined the system gives an audible chime and written warning[1, 28].

Ford’s Driver Alert System uses a forward facing camera to detect and track the lanes of the road from which indicators like lane departure can be estimated. After reaching a certain level of drowsiness a written alert is provided [1, 28].

The car company BMW has the system Attention Assistant. The 2013 version uses steering wheel patterns as its major fatigue indicator as well as other vehicle control factors. In cooperation with the supplier Denso International America, BMW introduced an image-based system to analyse the driver’s face by using a camera. In this case the main indicator is the analysis of eye-closure parameters, such as the widely accepted parameter PERCLOS, to identify drowsiness. However, BMW states that video-based methods to analyse facial expressions still cause many problems and that most of the car manufactures are using lane tracking instead [28].

The automotive parts manufacturer Aisin from Japan revealed an image based driver drowsiness detection system in 2012. It analyses facial expressions like eyelid movements and eye movements. It contains indicators like average blinking intervals, average eyelid closure duration, maximum eyelid closure duration, blink frequency, average eyelid closure velocity, average eyelid opening velocity, average eyelid position and integrated eyelid closure duration. With the help of these parameters the system classifies five stages of drowsiness: not drowsy, a little drowsy, drowsy, considerably drowsy and very drowsy. This system will soon be used by Toyota [30, 31]. The company Seeing Machine and its product faceLAB tracks head and face as well as eye, eye-lid and gaze. To track these facial expressions, it utilises a non-intrusive image-based system. It determines potential drowsiness indicators like blinks, PERCLOS and eyelid parameters. The technique is based on "real" eyelid position by using a multi-camera system and not on methods which use infrared. It is well established to be used in a driving simulator but faces problems in real world conditions, which mainly occur due to the different lighting conditions, in which the system has to work. Another product, the Driver State Sensor, is mainly used in the mining industry and works with two IR sources, a camera and a processing unit. The drowsiness indicators that are used by this device are the analysis of eye-lid parameters [1, 10, 12, 32-35].

The company Smart Eye AB tracks and detect facial features like face/head movement and eye movement. However, it does not provide the ability to estimate drowsiness [1, 10, 12, 36, 37].

In Summary, the commonly used main fatigue indicators by research groups and manufactures are: eyelid movement especially PERCLOS and driving/vehicle behaviour especially steering wheel patterns and lane deviation. In contrast, car- and automotive parts manufacturers use mainly driving/vehicle behaviour (in particular steering wheel patterns and lane deviation) as the main fatigue indicators. Eye parameters such PERCLOS, eye closure speed, blink rate and amplitude velocity ratios appear in the literature to be the most widely accepted parameters for drowsiness monitoring; showing the possibility to get the most accurate results (see chapter 1.1.2). Image-based technologies appear to be the gold standard to detect driver drowsiness in a non-intrusive way.

On this basis, drowsiness detection systems based on video analysis are discussed further. The basic requirements of the image-based detection system are the non-intrusiveness, real-time operation, tracking of the pupil, possible further development, widely used in research and robustness to different light conditions. A non-intrusive system does not disturb the driver and is more likely to reach a high acceptance. It is therefore important that the driver, for instance, does not have to wear glasses or an interaction with the system is not constantly necessary. A real-time operation solution guarantees real-time feedback, which is essential. The system has to track the pupil in order to estimate further important drowsiness indicators like blink-rate, PERCLOS and velocity of the eyelid movements (see chapter 1.1.2.). To ensure, that the system can cope with real-world driving conditions, operation under different light conditions, such as complete darkness, direct sunlight or ambient light, is essential. On the basis of the overview about noteworthy drowsiness detection systems (see Appendix Error: Reference source not found) four relevant products are selected, which meet the mentioned basic requirements above. The suitable products are shown detailed in appendix Error: Reference source not foundError: Reference source not found:

Seeing Machines - FaceLAB 5

Seeing Machines - FaceAPI

SmartEye - AntiSleep 3.3

SmartEye - Pro 3D Eye Tracking

Evaluation of drowsiness detection systems based of image-analysis

The goal of the following chapter is to offer an overview about relevant approaches to develop a drowsiness detection system. The first sub-heading gives an adequate overview about different methods to track the human eye, which provide determined information about the pupil, therefore making further estimations about several eyelid states realizable. Subsequently, the preferred method, and in this regard existing solutions, are compared and an optimal approach to develop a drowsiness detection system is suggested.

Image Based Methods to detect Eyelid States

To measure eye and/or eyelid movement four major methods can be used: Electrooculography (EOG), scleral contact lens/search coil, Photo-Oculography (POG) or Video-Oculography and video based methods. [38-40]

Electrooculography is a technique where electrodes are placed around the eye to measure an electrical field that depends on the movement of the eyes. The major benefits are that the method works also with closed eyelids, glasses and contact lenses as well it is robust to head movements. The disadvantages are its intrusiveness, non-ability for point of regard measurements and sensitivity to the electro-magnetic field. [38-40]

The most precise method to measure eyelid movements is by using special contact lenses, on which are either placed a mechanical or an optical object. Those reference objects can be for instance reflecting phosphor, line diagrams or wire coils, of which the latter is the most popular method. The technique is very accurate, has a high time resolution but it is extremely intrusive and not comfortable to use. [38-40]

Photo-oculography or video-oculography describes several methods to determine the position of the eye, for instance through: pupil tracking, limbus tracking and corneal reflections due to a light source. For example, limbus-tracking is realised by employing photo-diodes close to the human eye. [39]

Video-based methods usually capture the image through a camera, which is then processed in real-time to detect and track eye features like pupil, limbus or reflexion images. Limbus tracking utilises the high contrast between the iris and the sclera. This technique has a high horizontal but a bad vertical accuracy due to the overlapping from parts of the iris by the eyelids. In general, eye trackers can be divided into two groups: head-mounted and remote eye trackers. Due to the intrusiveness of head-mounted ones remote eye trackers are the better choice for drivers. Most of the eye-trackers work with light sources (mainly active IR) instead of using natural light to ensure constant light conditions and therefore a higher quality of the image. Using ambient light results in lower contrast images and a higher interference by light changes, but can handle direct sunlight better. Most of the infrared radiation spectrum is invisible to the human eye and operates therefore unobtrusively. The light sources in front of the eye generates different reflexions depending on their positions. Those reflexions are called Purkinje images and are applied for different tracking methods. There are two illumination options, which are used in an eye-tracking system: the dark pupil and the bright pupil technique. The dark pupil image occurs if the light is positioned off the axis and the bright pupil image results if the light is placed on the axis (see Figure ).

Figure General idea of an bright and dark pupil image [41]

The advantage of the bright pupil image (see Figure ) is that it has a higher contrast between pupil and iris and the pupil area is therefore easier to detect, whereas the dark pupil technique is more robust to the ambient light. With both techniques, a glint appears due to the reflexion on the cornea and gives information about the gaze. The cornea is physically different between individuals and for this reason the appearance of the glint varies from person to person. A calibration of the eye tracker is then essential, which can be avoided by using two or more cameras or more than one light source to generate different glints. The estimation of the glint can be mainly influenced by users who wear contact lenses.

Figure Images of a bright pupil (a) and a dark pupil (b) [41]

Visual clues have an important advantage that they are non-intrusive and an interaction between driver and system is not necessary. [38-40]

After detecting the pupil and the glint, drowsiness indicators such as gaze, PERCLOS, blink-rate and eyelid movements can be estimated. Further information how to calculate these indicators can be found in [38-40]. An approach on how to estimate the states of eyelids is described in [41] and [42].

Overview over Eye and Face-Tracker Solutions

As described in chapter 1.1.4.1 there are four major methods to detect and track the pupil. Professional eye-trackers use mainly image-based techniques, which are intrusive and widely used in the research field. First of all, it is important to gather information about existing eye-trackers and how they are adaptable for this study. The potential eye-tracker has to meet three major requirements: 1) The system has to operate with infrared light, 2) it must support a high resolution and 3) a high frame-rate. An eye-tracker that uses infrared is capable to solve night issues and copes well with varying light conditions. As a result, the image of the eye is sharper, the localisation is more accurate and therewith the entire tracking process is more robust, which makes the system suitable to work well under real-world conditions. According to [43] a high resolution of ≥1280x1024 is required, which results in better eye localisation and less noise. In addition, a suitable resolution results in a higher spatial detection and fulfils the qualifications to detect saccades and slow eye movements. To detect fast eyelid movements at about 100ms, a frame-rate at about 500Hz is necessary; this has been discussed to be in-line both with research conducted by [44, 45] and with the sampling frequency commonly used in EOG recording. However, previous research has suggested a frame-rate of 200Hz is sufficient to create a signal of eye-blink movements that matches the simultaneously recorded EOG [46, 47]. Other requirements, which are discussed as well, are:

Price

Allowable Head Movement

Operating Distance

Sensor Technology

Glasses compatibility

Output-Data

Necessary adaptation of the system

In Table the following commercial products or research groups are compared: Seeing Machines –FaceLab 5, Seeing Machines –FaceAPI, Smart Eye –AntiSleep, Smart Eye –Pro 5.9, SMI – Red 500 (250, 120, 60), SR Research -Eye Link II (Remote option), Tobii - TX300, VisonTrak – ETL 600 and ITU GazeGroup.

According to Table there are three different options for the development of a drowsiness detection system, which is based on video analysis: 1) Adaptation of a commercial drowsiness detection system, 2) adaptation and further development of a basic eye-tracker and 3) the development of an original drowsiness detection system. The development of an original drowsiness detection system has three major advantages: 1) the independency of foreign intellectual property, 2) the flexibility of creating an original system and 3) the price. Commercial products are usually protected by patents and changes of the systems for research purposes are difficult because the respective companies supply limited information about the system [38]. With regard to Table , the most promising approach is the further development of the open-source project from the ITU Gaze Group. It is a non-commercial and free product and meets the three major requirements explained above (operation with infrared light, high resolution and high frame-rate).

Req. / Product

Seeing Machines -FaceLab 5

Seeing Machines -FaceAPI

Smart Eye -AntiSleep

Smart Eye -

Pro 5.9

SMI - Red 500

(250, 120, 60)

SR Research -

Eye Link II (Remote option)

Tobii - TX300

VisonTrak - ETL 600

ITU GazeGroup

Price

$35 000

Development license

Unknown

$30 000

No E-mail reply

CA$40 000

$50 000 hardware + $10 000 software +GST

US$30 000 (additional costs for remote and high speed option)

Open-source

<$5000 for hardware

Frame-Rate

60Hz

60Hz

60Hz

120Hz (2-4 cameras)

60Hz, 120Hz, 250Hz, 500Hz

250Hz, 500Hz

300Hz

240Hz

Up to 500Hz

Allowable Head Movement

35x23cm

35x23cm

20x15x20cm

40x40x30

40x20 at 70cm distance

22x18x20cm

37x17cm

30x30x30cm

Unknown

Operating Distance

<60cm

<60cm

Normal driving positions

30-300 cm - Adjustable with lenses and positioning of cameras

60-80cm

40-70cm: EyeTracker

50-80cm

45-90cm

Unknown

Sensor Technology

CMOS

CCD

Resolution

≥1280x1024

≥1280x1024

640x480

640x480

≥1280x1024

≥1280x1024

≥1280x1024

1500x2000 (60Hz)

≥1280x1024

Brightness condition

Works in light and dark environments (maybe with less functionality

Robust to lightening changes

Works in sunlight and darkness

"handles all variations of natural illumination conditions"

"works in sunlight as well as in darkness"

No reply

Direct sunlight poses issues

Most light condition but no direct sunlight

Most light condition but no direct sunlight

Unknown

Glasses compatibility

Able to work with subjects wearing sunglasses, contact lenses, and most eye-glasses

Robust to glasses

highly robust to all natural illumination conditions in automotive applications

insensitive to the ambient lighting

"Works with most glasses and contact lense"

YES

YES

YES

Unknown

Output-Data

Head position

Head rotation

Eye position

Eye Rotation

Eye gaze local

Eye gaze global

Pupil diameter

Eye vergence distance

Saccade events

Blink events

Blink frequency

Blink duration

Eyelid aperture

Eyelid behaviour

Lip behaviour

Landmarks: Head position

Head rotation

Eye position and other

Lip behaviour

Head position, head orientation, gaze direction, eyelid-opening

Head position, head rotation, gaze direction, eyelid opening, pupil diameter, blinks

Gaze position

Pupil diameter

Pupil position

Tracking status

Eye image

Access eye position data

DataViewer:

eye event position; eye sample trace visualization; gaze position overlay; eye sample, fixation, saccade;  mainly eye gaze

Eye position, gaze point, pupil diameter, validity code

Gaze, pupil data

Gaze data, glint position

Necessary adaption of the system

Algorithm for additional fatigue indicators

Algorithm for additional fatigue indicators

Algorithm for additional fatigue indicators

Algorithm for additional fatigue indicators

Algorithm for main fatigue indicators

Algorithm for main fatigue indicators

Algorithm for main fatigue indicators

Algorithm for main fatigue indicators

Algorithm for main fatigue indicators, improvement of the open-source software, installation of the hardware

Summary

Expensive, frame rate too low for adequate analysis of eyelid movements

Mainly face-tracking, Development license available, frame rate too low for adequate analysis of eyelid movements

End-consumer product, frame rate too low for adequate analysis of eyelid movements

Expensive, medium-low frame-rate, moderate operating distance (manual changing necessary), supports 2-8 cameras

No email reply, high frame-rate allows advanced analysis of eyelid movement, restricted operating distance

Expensive, high frame-rate allows advanced analysis of eyelid movement, restricted operating distance

Very expensive, medium frate, restricted operating distance, well known eye-tracker

Expensive, medium frame rate, restricted operating distance

Inexpensive, not as accurate as commercial ones, set up time uncertain, no support, individual design possible

Table Matrix over Eye and Face-Tracker Solutions

System Proposal to detect driver drowsiness

As discussed in 1.1.4.2 the optimal solution for developing a drowsiness detection system, is to adapt an existing open-source eye-tracker. Various projects have already been undertaken to develop remote low-cost eye trackers. Such projects provide information about how to build an eye tracker off the shelf and also offer suitable software, which is mainly open source [38].

One promising project is an eye tracker from the ITU GazeGroup.The eye tracker provides tracking of the pupil and the glint and gives information how to set up the needed hardware components. The system (30Hz, monocular) was compared by [48] with the commercial eye tracker Tobii (T60, binocular, 60Hz) and one of the main results is that the T60 systems is twice as accurate in the estimation of the gaze as the ITU GazeTracker. The ITU Gaze Tracker was also evaluated in [49] and found to be accurate and robust.

However, open source products do not offer fully established solutions. It is still necessary to develop an appropriate system around the open-source software. In general, the following steps are necessary to set up an original system, based on an open-source program:

Deploying Hardware components (see Figure ):

Computer (typical desktop computer) [48]

Remote Camera without IR blocking filter

Several lenses

IR lamps

Configuration of the open source eye tracking software

Calibration of the system

Adjustment of the system

IR LED 1

IR LED 2

Camera

Figure Suggested System

In summary, the objective of the system is to estimate eye-movement parameters in a non-contact method, where eye-parameters such as PERCLOS, Eye closure speed and blink rate can be determined, because they appeal in the literature to be the most widely accepted parameters; showing the possibility to get the most accurate results. The implementation of the software is based on the combination of the open-source project ITU Gaze Group and a custom built solution. The hardware mainly consists of a PC, a camera and IR lights (see Figure ). Table Overview of required Hardware illustrates the required hardware in more detail. An explicit listing of suitable cameras is shown in appendix Error: Reference source not found.

High speed camera

High speed & resolution camera

Point Grey Camera

$1200 (Point Grey)

Visible light filter

IR passing filter (800nm)

$50

Lens

5-15mm

$200

IR illumination

Two infrared light-emitting diodes (820nm)

$100

Computer

Recommended System Configuration based on the Point Grey camera:

n OS—Windows 7 32- or 64-bit

n CPU—Intel Core i3 3.1 GHz or equivalent

n RAM—2 GB

n Video—NVIDIA GeForce6 128 MB RAM

n Ports—PCIe 2.0 compatible host controller with

USB 3.0 connector

n Software—Microsoft Visual Studio 2005 SP1

$0 (already available)

≈$1500

Table Overview of required Hardware

The described system is a basic approach and can be adapted at a later stage if needed. The number of cameras can be varied. More cameras increase the horizontal field of view without loss of resolution, which is important because the eye tracker requires a certain pixel size to analyse the eye [50]. In addition, the use of more than two infrared lights makes a user calibration redundant and therefore easier to use.

Further reading can be found in: [49], [48]. [51], [52], [53], [50], [54]