Eye Tracking Making Menu

Published Date: 02 Nov 2017

SELECTION WITH YOUR EYES

JIECAI HE

ABSTRACT

As more smartphone owners start adopting tablets,

we need to _nd a new way to interact with our tablets

since they have larger screen and greater computing

power than smartphones in general. We propose we

use eye tracking as an alternative input method in ad-

dition to multi-touching and voice commanding. Since

users are constantly looking at screen before they touch

or say something, if we can recognize users' intention

beforehand, we can avoid the need to touch or voice-

commanding the device. We track the user's eye move-

ment using the front camera then using machine learn-

ing algorithms to locate the pupil position to _gure out

what position the user is looking at and select the corre-

sponding menu. We present a prototype implementation

of our system using Eee Pad Transformer TF101, since

a similar project has implemented their prototype on

iPhone.

Keywords

Human-Phone Interaction, Machine learning, Android,

Tablet, Eee Pad Transformer

1. INTRODUCTION

In this project, we like to use eye tracking to enable

a user to select certain menu item using eye movement.

We implement our prototype on Android tablet - Eee

Pad Transformer TF101 using JAVA. We will use front

camera to locate the face _rst then locate the two eyes

within the face, then locate the pupil, _nally infer the

area of the screen the user is looking at using the found

pupil position.

2. DESIGN

Our algorithm will exploit the fact that pupil is the

blackest region in eye area. However, heavy black makeup

or strong black eyebrow will compromise this assump-

tion. However, if the eye detection algorithm property

centers the eye area in a square, we can eliminate the

top and bottom part of the square, so we avoid wrongly

detecting area with black makeup or eyebrow as pupil

Figure 1. [1] Pupil is the blackest cir-

cular region within eye area

.

position. First, I will describe the procedure used to

locate the position of the menu item which the user is

looking at.

2.1 Locate Face

We build our project on top of the OpenCV face-

detection demo [2]. The demo is capable of doing face

detection. The demo uses Local Binary Patterns (LBP)

cascade classi_er to do face detection. "LBP features

are integer in contrast to Haar features, so both train-

ing and detection with LBP are several times faster then

with Harr features" [3]. However, the speed also depends

on the quality of the dataset and the parameters used

during training process. The trained classi_er cascade is

stored in an XML _le called 'lbpcascade frontalface.xml'

under nfaceô€€€detectionnresnrawn. Haar-like (including

LBP) classi_er can be used to detect "blocky" features

such as eyes, mouth, face, cars, etc. but it works less

well with object where its most distinguishing charac-

teristic is its outline shape [4, p. 510]. We can train

our own classi_ers to detect other objects by follow-

ing the procedure described on [4, p. 513] given we

have gathered enough positive and negative examples.

In Android, we _rst need to load our classi_er _le into

CascadeClassifier object. Once we did it, we call

mCascade.detectMultiScale(mGray, faces, 1.3, 2, 2

, new Size(faceSize, faceSize), new Size());

to detect faces. The detected faces will be stored in

MatOfRect faces. The next step is to locate eyes within

the detected face.

2.2 Locate Eyes

We use Harr cascade classi_er to locate two eyes within

the detected face. We can _nd _le called 'haarcascade eye

tree eyeglasses.xml' from OpenCV Windows distribu-

tion. We then transfer the _le to directory 'nfaceô€€€detectionn resnrawn' under our Android project. We can then load

the _le into CascadeClassifier object and call the

function detectMultiScale to detect eyes.

1

2 JIECAI HE

Figure 2. Notice that the pupil posi-

tion on the left eye is o_ since the algo-

rithm we used for left eye will degenerate

when there is a reection on pupil.

mEyesCascade.detectMultiScale(mFace, eyes,

1.2, 2, 2,new Size(minEyeSize , minEyeSize),

new Size(maxEyeSize, maxEyeSize));

The detected eyes will be stored in MatOfRect eyes.

The next step is to locate pupil within the detected eye.

2.2 Locate Pupil

We will use the following procedure to detect pupil

position.

2.2.1 Turn the color image containing the eye into

gray scale.

Mat matSecondEyeGray = mGray.submat(new Rect

(new Point(r.x + e.x, r.y + e.y), new Point

(r.x + e.x + e.width, r.y + e.y + e.height)

));

Note that mGray is an object of Mat which is an ar-

ray structure in OpenCV used to hold a frame of video

stream. In above code, we extract the subsection con-

taining the eye from the entire video frame using the

location of the detected face r and eye e.

2.2.2 Eliminate unnecessary pixels above the eye area.

As we mentioned before, our assumption that pupil is

the darkest area around eye may not be true when the

area above eye is covered with black strong makeup or

thick eyebrow. To avoid such mistake, we can turn the

area above eye into less-dark area by manipulating the

pixel intensity of the mat object.

// Eliminate the top part (above the eye)

byte buffSecondEyeElimination[] = new byte[(int)

(matSecondEyeGray.total() *

matSecondEyeGray.channels())];

matSecondEyeGray.get(0,0,

buffSecondEyeElimination);

for (int i = 0; i < matSecondEyeGray.total() /

matSecondEyeGray.rows() * 8; i++) {

buffSecondEyeElimination[i] = Byte.MAX_VALUE;

}

matSecondEyeGray.put(0,0,

buffSecondEyeElimination);

First, we create a byte array to hold pixels. Next, we

put entire pixels of the mat object (square shape) into

the array. We then change the pixel of the _rst 8 rows

of the square into less-dark value. Finally, we put pixels

in the array back into the mat object to allow further

manipulation.

Figure 3. The picture is in gray scale

and the area above the eye is less-dark

than before

2.2.3 Use Histogram Equalization to increase black

and white contrast.

We will use Histogram Equalization [5] to increase

black and white contrast, so pupil will become more

dominant in turns of pixel intensity.

Imgproc.equalizeHist(matSecondEyeGray,

matSecondEyeEqualized);

Figure 4. The picture has higher con-

trast than before.

2.2.4 Invert pixel intensity

Next, we need invert pixel intensity. "Black becomes

white, white comes black". So we can apply erosion in

next step.

Core.bitwise_not(matSecondEyeEqualized,

matSecondEyeEqualizedInverted);

EYE TRACKING: MAKING MENU SELECTION WITH YOUR EYES 3

Figure 5. "Black becomes white and

white becomes black"

2.2.5 Erosion

We will use erosion to "eat away" the white area. Ero-

sion is good for eliminating noises in the picture (small

area with strong pixel intensity.)

Imgproc.erode(matSecondEyeEqualizedInverted,

matSecondEyeEqualizedEroded, new Mat(),

new Point(-1, -1) , 2);

Notice the third parameter speci_es the structure el-

ement used to erode. In this case, we use default value.

We may _nd some speci_c structure element that works

better with circular object.

Figure 6. White area indicates the

darkest area in the original picture

2.2.6 Threshold the picture into binary

In OpenCV, we can either apply a _xed or adaptive

threshold to a picture. In this case, we pick _xed thresh-

old since it is easier to use.

Imgproc.threshold(matSecondEyeEqualizedEroded,

matSecondEye, 200, 255, Imgproc.THRESH_BINARY);

Note that 200 is the thresh value we used.

Figure 7. Binary picture

2.2.7 Find contours

We will use Imgproc.findContours to _nd contours in

the binary picture. Once we found it, we will trans-

verse through each one and use the area speci_ed by the

contour to _nd the average pixel intensity under that

area in the original gray-scale picture. We will then use

the maximum average pixel intensity to threshold the

eroded picture again. We pick the contour with largest

area to be the one containing pupil.

Figure 8. We isolate the pupil into an

isolated contour

2.2.8 Locate center of the pupil

This is the most critical part of our project. My _rst

attempt was to apply Opencv's Imgproc.moments func-

tion to the isolated contour to _nd the center of the

mass. However, the result is poor when there is a reec-

tion on the pupil as you saw on the left eye of the Figure

2. We should "Before thresholding and after threshold-

ing though, a new erosion is applied on the ROI in or-

der to remove additional noise, reections of light on

the pupil and so on" [6]. This method requires me to

extract the area bounded by the contour from our gray

scale picture, apply erosion to remove noise or reec-

tion on the pupil then increase the boundary (extract

a larger region from the gray-scale) to include the en-

tire pupil, the value returned by moments should give

us the correct coordinate of the center of the pupil.Due

to the unexpected di_culty 1, I didn't implement this

method. I simply take the center of the bounding box

of the contour as the center of the pupil.

Figure 9. Geometric center of the con-

tour is the center of the pupil

2.2.9 Locate the menu item

To locate the menu item, we need to _gure out whether

the user is looking left, right, top, bottom or center. To

_gure out the horizontal position, we can measure the

length of the horizontal line (going through the center of

the pupil) talking on the while color on both side of the

pupil. If ratio of two lengths is close to 1, then user is

looking at the center. If ratio is very small or very large

then the user is looking left or right. It's more di_cult

to _gure out the vertical position, since we don't have

white area to work with. We can try to _t the pupil to

a circle, then the center of the _tted circle will give us

a good indication of the position of the pupil. However,

1While waiting my Eee pad replacement charger to arrive, I

used an unstable Lab power supply to charge my tablet. I guess

that toasted the connector, after that computer won't recognize

Eee pad as external device. This means I can't use Android Log()

function to do debugging. The only way I can see the value of a

variable is to write on the mat object. Too much to bear!!

4 JIECAI HE

_tting to circle (Circular Hough Transformation) may

not be computationally e_cient. In our case, due to the

same di_culty, we simple use the position of the pupil

in the bounding box as the only cue to _gure out what

the user is looking at. If the pupil coordinate is close to

the left boundary of the bounding box then it is looking

at right etc.

Figure 10. The coordinate of the pupil

decides which menu item the user is look-

ing at.

Depending on which area the user is looking at, the

corresponding menu item will be clicked. We can further

robust the activation mechanism by using winking [7].

3. PROGRAMMING

One thing I have to say is that it is not easy to pro-

gram using OpenCV and Android. I will mention a few

problems I encountered.

3.1 Redesign User Interface

There are two ways to design user interface in Android.

The _rst one is to code the user interface using java.

The other is to use XML _le to specify the layout. We

will XML _le to design our layout. To align a few View

objects horizontally, we do the following.

<LinearLayout

android:layout_width="wrap_content"

android:layout_height="0px"

android:layout_weight="0.01"

android:orientation="horizontal" >

<view class="View1"

android:id="@+id/view1"

android:layout_width="0px"

android:layout_height="wrap_content"

android:layout_weight="0.01"

/>

<view class="View2"

android:id="@+id/view2"

android:layout_width="0px"

android:layout_height="wrap_content"

android:layout_weight="0.01"

/>

</LinearLayout>

Notice that we specify the LinearyLayout's

android:orientation to be horizontal.And any view

contained in the LinearLayout has android:layout width

set to be 0 px and and android:layout weight set to

be 0:001. By doing this, these two view will each take

on 50% of the available horizontal space. To align two

view vertically, we do

<LinearLayout

android:layout_width="wrap_content"

android:layout_height="0px"

android:layout_weight="0.01"

android:orientation="vertical" >

<view class="View1"

android:id="@+id/view1"

android:layout_width="wrap_content"

android:layout_height="0px"

android:layout_weight="0.01"

/>

<view class="View2"

android:id="@+id/view2"

android:layout_width="wrap_content"

android:layout_height="0px"

android:layout_weight="0.01"

/>

</LinearLayout>

By nesting LinearLayout vertically and horizontally,

we can easily create the layout we want.

Figure 11. Combine LinearLayout

horizontally and vertically to create so-

phisticated layout.

3.2 Display Mat on view

To display portion of the video frame on View ob-

ject, we need to convert the Mat object into Bitmap

object and call View's postInvalidate() in another

thread. We call this function in another thread, so An-

droid's main thread won't become unresponsive. The

postInvalidate() will call View's onDraw() to re-draw

EYE TRACKING: MAKING MENU SELECTION WITH YOUR EYES 5

the view. Since we can't pass the Bitmap as

postInvalidate()'s parameter to View object when we

call postInvalidate(). So we need to create a

public, static variable on Activity class to hold

the Bitmap object. So every time, a View's onDraw()

is called, it can use the Bitmap contained in this vari-

able to update the view. The requirement of accessing a

static variable in View object makes this View object

into Static class. So we have

private static class leftEyeView extends View{

public leftEyeView(Context context,

AttributeSet attributeSet) {

super(context, attributeSet);

}

public void onDraw(Canvas canvas) {

super.onDraw(canvas);

if (LeftEye != null) {

canvas.drawBitmap(LeftEye, 0, 0, null);

}

}

}

In this case, LeftEye is the public static Bitmap

in Activity class which got updated each frame. When

we specify layout in XML _le, we need to use a special

syntax to access this static class.

<view class="

org.opencv.samples.fd.FdActivity$leftEyeView"

android:id="@+id/viewLeftEye"

android:layout_width="0px"

android:layout_height="wrap_content"

android:layout_weight="0.01"

/>

4. FUTURE WORK

The algorithm we used to locate the center of the

pupil and the menu item which the user is looking at is

trivial. however, this is the best I can do. '.'