Human Emotion Using Smart Sensors

Published Date: 02 Nov 2017

SMART SENSORS

ABSTRACT

Emotion recognition has become an important subject when it comes to human-machine interaction. A smart sensing system, which would help in detecting human emotions based on information from physiological parameters obtained from sensors, has been designed. The signals are continuously obtained from a heart rate sensor, skin temperature sensor, and a skin conductance sensor. After amplifing and filtering of the signals from the sensors are done it are processed in the microcontroller and transmitted wirelessly using ZigBee technology. The signals which are received from the system are displayed and stored on the computer where they are analyzed visually for obvious patterns. The four basic emotions observed in this project are happy (excited), sad, angry and neutral (relaxed). K-means clustering algorithm has been used to cluster data into four groups (emotions) to automatic recognition of human emotion. A graphical user interface (GUI) has been designed to communicate with the hardware as well as display real-time emotion(s) for the monitored period. The developed system has shown good results in monitoring the physiological parameters.

Keywords: Human emotion, smart sensors, heart rate, skin temperature, skin conductance, zigbee, K-means clustering algorithm.

INTRODUCTION

Recent researches have been focused on improving quality of human life in terms of health by designing and fabricating sensors which are either in direct contact with the human body (invasive) or indirectly (non-invasive). These sensors based system will also have a positive impact on the annual medical cost and the health management system[1].Emotions play critical rolein rational and intelligent behaviors. It is a mental state that does not arise through free will and is often accompanied

by physiological changes. These changes need to be monitored as they contain information about different

types of emotions which will assist in understanding behaviors[2].

The aim of this paper is to design a non-invasive system which will be capable of recognizing human emotions based on measuring physiological parameters of a human body using smart sensors.

The design ideas and specification of this system consists of the following features.

Affordability: The system has to be cost effective.

USB/Wireless compatibility:The system should connect to the computer wirelessly or through USB.

Reliability: The system has to be reliable.

User friendly: Both the systems hardware and software has to be user friendly.

II. LITERATURE SURVEY

In the past different approaches and methods have been used to detect and evaluate human emotions. The most common ones include the use of electroencephalogram (EEG) signals, facial expressions, speech, body gestures, textual information and physiological signals.

Emotion Recognition using EEG:

EEG signals have been used to analyse the nervous system to provide information about different types of emotions. [3]

Emotion Recognition using facial expression:

Cameras and image processing techniques are used to detect changes in facial expressions using external stimulus to excite specific emotions. [4]

Emotion Recognition using speech:

Voice recognition systems monitor the tone of speech and hence providing information about an individual’s emotion.[5]

D. Emotion Recognition using text:

Textual emotion recognition is based on words or sentences used in a developed chatting system.[6]

E. Emotion Recognition using body movement and gesture:

This method evaluates emotion based on specific actions or dances assigned and naturally occurring, which is one of major drawbacks.

F. Emotion Recognition using Physiological signals:

In this paper, an emotion recognition system is developed using physiological signals. These signals are obtained from a heart rate sensor, a skin temperature sensor and a skin conductance sensor and stored for data analysis and feature extraction. The four basic emotions considered in this paper are happy (excited), sad, angry and neutral (relaxed).

OBJECTIVES

The main objectives of this paper are,

To design a real time monitoring system, capable of evaluating four basic emotions i.e. happy, sad, angry and neutral, using low cost and non-invasive physiological sensors.

To design a system which should be capable of monitoring data in a comfortable and unobtrusive manner, with the ability of wirelessly communication.

To design a systems software and hardware should be easily upgradeable and compatible with new sensors and latest operating systems respectively.

The system should be capable of wirelessly communicating with a computer in order to monitor data without being attached to the computer.

SYSTEM DESIGN

The system consists of transmittersection and receiver section. In transmitter section physiological sensors, signal conditioning circuit, a C8051 microcontroller and Zigbee module is present. In receiver section zigbee coordinator and computer is present. The data from the system is wirelessly transmitted to a computer where it is displayed and stored. Figure 1& 2 shows the block diagram of the transmitter and receiver section of the proposed system.

FIG1: Block diagram of transmitter section.

FIG 2: Block diagram of receiver section.

A. Heart Rate Sensor:

The most common heart rate variability at rest can be due to serious heart problems, respiratory problems and emotional imbalance i.e. stress, panic attacks, anxiety and depression.

In this paper we are interested in finding relationship between heart rate and emotions. The heart rate sensor used in this paper is based on the concept of photoplethysmography (PPG). PPG is an optical measurement technique that is used to detect blood volume changes in the micro vascular bed of tissue. The reason for using this approach is because it is simple, non-invasive, compact and reliable for continuous monitoring of heart rate. The theory behind the heart rate sensor is that the amount of light detected by the phototransistor varies with the blood flow as light is only absorbed by the haemoglobin in the blood. Therefore when an infra-red light is directed onto a human finger, the light is detected as pulses by the phototransistor which is the measure of heart rate. PPG can be used in two different ways. One way is to have the light detector and light source on the same side of the skin, this is called reflectance PPG. The other way is to have light detector and light source are placed across the skin, this is called transmittance PPG[7].

FIG 3: For photoplethysmography, increased blood decreases received light in (a) transmission mode and

(b) reflection mode.

In this paper we use transmission PPG as it is more reliable and resistant to movements and external noise. Figure 3 shows the model of transmission PPG and reflectance PPG. The custom made heart rate (HR) sensor consists of an infra-red LED (OP180) with a wavelength of 940nm by Optek Technology and an infra-red phototransistor (SDP8406). A low power quad operational amplifier (LM324) is used for the amplification of the signals. The heart rate sensor requires 3.3V to operate which is provided by the microcontroller. The output from the HR sensor is a digital signal which is input into the digital port of the microcontroller.

B. Skin Temperature Sensor:

Continuous monitoring of body temperature is very important. Generally human skin temperature is between 32-35ºC . However, there are many causes of variation from these values. The most common reasons for changes in skin temperature at room temperature include fever, malnutrition, physical exertion and physiological changes. In this project we wanted to observe the changes in skin temperature and find the relationship between these changes and emotions.

Skin temperature is measured using DS600 analog-output temperature sensor by Maxim - Dallas semiconductor. The reason for choosing this sensor is because it is in-expensive, has low power consumption and has an exposed pad which will be in contact with the human skin for continuous temperature monitoring. The pin configuration of the sensor is shown in figure 4.

FIG 4: Pin configuration of DS600

The DS600 requires 2.7-5.5V to operate which is provided by the microcontroller. It provides an accuracy of ±0.5˚C over a range of -20 to 100˚C [8] . The output from the sensor is an analogue voltage which is proportional to temperature in ºC and is given by the formula.

T (ºC) = (Vout - VOS) / (ΔV/ΔT)(1)

Where,

VOS = DC offset, 509mV

ΔV/ΔT = Typical output gain, +6.45 mV/ ºC

VOS = DC offset, 509mV

ΔV/ΔT = Typical output gain, +6.45 mV/ ºC

The output characteristics of DS600 is shown in figure 5.

Fig 5: Output voltage characteristic of DS600

The output from the skin temperature sensor is input into the analogue port of the microcontroller for processing.

C. Skin Conductance Response (SCR) Sensor:

Human skin has electrical properties that change relatively quickly and are closely related to psychological and physiological processes. Changes in electro-dermal activity (EDA) and skin conductance are related to changes in eccrine sweating gland which, in turn, are related to activity in the sympathetic branch of the autonomic nervous system (ANS). Therefore, skin conductance has become an important tool to help find human emotions and motivation which is why we decided to choose skin conductance response as one of our signals for emotion recognition.

The skin conductance response sensor is based on simple voltage divider rule. Figure 6 shows the circuit diagram of the SCR sensor. R1 is the electrical resistance of the human body where the human body acts as a large resistor. Two electrodes (A and B) stay in contact with the human skin via the middle and ring finger. C1 is a 1μF capacitor, and R2 is a 68KΩ resistor. As we know from voltage divider formula the Vout is only dependent on the value of R1 as R2 and C1 are kept constant.

The output from the SCR sensor is a voltage value which is input into the analogue port of the microcontroller for processing.

Fig 6: Skin conductance response circuit

D. Microcontroller

The C8051 F020 is a fully integrated mixed – signal system-on-a-chip microcontroller. It has total of 64 general purpose ports I/O pins. The lower ports (P0-P3) are both bit- and byte- addressable. The upper four ports are only byte-addressable GPIO pins.

The features of the c8051 are as following:

High speed pipelined 8051 compatible CIP-51 microcontroller core.

In system full speed, non-intrusive debug interface(on-chip).

True 12-bit 100 ksps 8-channel ADC with PGA and analog multiplexer.

64k bytes of in-system programmable flash memory.

External data memory interface with 64k byte address space.

E. Analog signal processing

Temperature sensor and skin conduncance sensors produce a analog signals as output. These signals are given to the ADC0 of the microcontroller. ADC0 consists of 9-channel analog multiplexers, programmable gain amplifier and 12- bit successive approximation register ADC. Three channels are used for measuring skin temperature , room temperature and skin conduntance. Timer 3 is configured to autoreload at a specific interval using SCR. The ADC make a repeated measurement in a rate of 50KHZ determined using Timer 3.

F. Communication

The system is designed to communicate with the computer wirelessly as well as using an RS232-USB[9]. The wireless communication is achieved by using Zigbee technology [9].

Zigbee is suitable for low rate data and secure networking. It has been widely used for monitoring applications.

It has a range of up to 40m. In this project we used XBee Series 2 OEM RF modules which operate within the ZigBee protocol and uses 2.4 GHz frequency band. It requires a supply voltage of 3V which is provided by the microcontroller and consumes power as low as 296mA.

v. HARDWARE MODEL

The developed system is in a shape of a box with a surface area of 220mm x 145mm which is slightly larger than an average hand. The microcontroller and signal condition circuits reside inside the box while the HR sensor, skin temperature sensor and SCR sensor electrodes are mounted on top of the surface in order to be in direct and easy contact with the hand. The system is designed to monitor data from the left hand which is shown below in figure 7.

Fig 7: 3D model of the surface of the system with thesensors and electrodes.

The signal from sensors are processed by C8051 microcontroller and sent wirelessly by the Zigbee router which is received at the other end by the Zigbee coordinator.

The coordinator is connected to the computer via the USB port. The data after being received is displayed using LabView and is stored as an excel file. The next step was to collect data from different subjects and observe the changes. This is explained in the results section below.

RESULTS

Once the reliability of the sensors was tested, we started collecting data from different subjects. This data was stored in order to help in finding obvious visual patterns and also use it for feature extraction. The results of BPM, skinconduntance data are collected from subjects in happy (excited), sad, relaxed (neutral), and angry states have to be done.

The skin temperature and the room temperature of the subjects were also monitored. The results proved that different emotions have effect on the physiological signals which can be visualised by continuous monitoring. The heart rate variability and skin conductance response were the dominant signals in this case. The SCR values showed that, the higher the relaxed state, the lower the SCR values get and vice versa which proves that the SCR values are directly related to the level of arousal.

FEATURE EXTRACTION

To develop an algorithm which would help in the automatic recognition of emotions. This requires collection of a large number of data, applying different clustering methods and extracting information related to various features. We looked into k-means clustering technique where we grouped the data into four clusters. K-means clustering is a method of clustering which concept is to partition ‘n’ observations into ‘K’ clusters in which particular observation belongs to the cluster with the nearest mean. The main aim of k-means clustering technique is to find out centres of natural clusters in the data using iterative refinement technique. In order to cluster the data, the training data collected from 14 individuals are used. Then the preprocessing, classification, regression, clustering and visualization of the collected data have to done. The data collected from subjects showing four emotions (happy, sad, neutral and angry) were combined together and grouped into four clusters using k-means clustering method.

CONCLUSION

In this paper, we have developed a real time emotion recognition system based on data provided by physiological sensors. Physiological sensors were found to be the best approach to recognize emotional changes, as they provided information about changes that take place physiologically and are out of the person’s control. The sensors have been integrated and placed on a surface of a designed box, for collecting data from the left hand. This design enabled an easy and comfortable data monitoring system.

IX. FUTURE WORK

For future improvement and development of the system, the use of additional physiological sensors such as pressure sensor, respiration sensors, and other sensors is required. In order to get more information from the data other clustering methods should be looked into which can help in improving emotion recognition rate.The system once fully developed will be capable of extracting basic emotions i.e. happy, sad, angry and neutral from the physiological signals. It can be integrated with a computer mouse which will help in evaluating emotional health of computer users in schools, universities and offices for a longer period of time.