An Adaptive Color Night Vision Scheme

Published Date: 02 Nov 2017

Abstract—Color night vision has been the debate of extensive research during the last ten years, primarily due to its importance in many real applications. In this research a brightness compensation system in vision images and videos based on Evolutionary Algorithms (EAs) was proffered. Enhanced Particle Swarm Optimization (EPSO) has been proved to be effective at finding optimal solutions of the proposed visioning problem by adapting the best global parameters of a novel extension to a local adaptive vision technique. As well the framework of the proposed system is being accurately developed and tested, and the mathematical analysis is mainly depends on the fitness score being developed, peak-signal-noise-ratio and the averaged brightness. Where the feasibility of the proposed system is compared with Differential Evolution (DE) and Artificial Neural Networks (ANNs). At all, the prototype of the system is envisaged to be applicable in many domains, and the avail of this systems leads to a so-called color night vision system.

Keywords- Mean Square Error (MSE), Peak Signal Noise Ratio (PSNR), Differential Evolution (DE), Enhanced PSO, Artificial Neural Networks (ANN).

Introduction

Vision system is an important subject that has been applied in many real imaging applications. The vision systems can be considered as the most powerful vision trend among different senses, since it can provide a highly significant deal for the intelligent interaction with the environment [1, 2]. For instance, in the night or under afterglow environments, it is hard to capture color information, which is very important for color discrimination. So, it is essential to highlight the importance of night vision systems in objects recognition since most models depend, directly or indirectly, on edge maps of input images for scene analysis and interpretation.

Practically all works in computer vision is based on the premise that the observer is immersed in a transparent medium. The color discrimination in biological visual systems is ascribed to perceiving the difference between the frequencies of light waves transmitted by various objects. The adaptation of the vision system to work satisfactorily under varying lighting conditions is a big issue to deal with. Thus if this system is in place, it will definitely help to develop many aspects of computer vision. So, more weights should be dedicated to the night vision systems which can help to develop the robots in all types and under all conditions. If the developed system is put to work under such situations, the public are to benefit from better services. The beneficiaries of this work are the researchers, physicians, surgeons, physicists, students, developers and manufacturers of robot vision systems.

Some common applications for night vision systems include tracking moving objects [2], video surveillance systems [4], traffic monitoring [5], silhouette detection [6], robots, mobile robots and navigation applications. Consequently, there has been an immense effort that has been directed toward vision systems. Despite this great effort, though adaptive vision system is still an unsolved problem and so far from being ripe, and actually the active research in the color night vision is still opening up to date. Accordingly, there is an insistence from researchers to solve this problem, and this can be effective to present and quantify the night vision systems. Alternatively, the existing techniques in computer vision are usually time consuming and may require high processing power, and sometimes are not well suited since they do not consider the conditions of low-light-level imaging [4]. However, if these techniques can be successfully replaced by a new technique that takes less processing time and less processing power, it would be a greatly superior technique.

A commonly encountered problem in color night vision systems is the variations in lighting and camera settings, for instance, when the lighting conditions are changed, and the location of the camera is fixed, the obtained images are not the same [1]. So it becomes a must for the vision system to be adapted to the possible lighting conditions prevalent in its environment. Take an example, mobile technology has been rapidly grown; therefore, there is an urgency to further up adding more features to the night vision systems, whereas, the conventional visioning techniques existing in mobile robots work on complex mathematical and statistical techniques. Also, more complex techniques that are used in computer vision cannot be used in mobile robots due to the existing constraints of consuming time and processing power.

In this paper, a novel approach is proposed to investigate the suitability to identify and recognize the gloomy objects under little illuminations, which can be useful for the assessment of night visions, and it can be applied to many imaging applications. This study is mainly depends on using Particle Swarm Optimization (PSO) [7, 8] for parameters estimation on the basis of objective evaluation function. Where the PSO is used as an evolution technique for extracting the color information from the night vision images based on some developed mathematical transformation equation.

All standard paper components have been specified for three reasons: (1) ease of use when formatting individual papers, (2) automatic compliance to electronic requirements that facilitate the concurrent or later production of electronic products, and (3) conformity of style throughout a conference proceedings. Margins, column widths, line spacing, and type styles are built-in; examples of the type styles are provided throughout this document and are identified in italic type, within parentheses, following the example. Some components, such as multi-leveled equations, graphics, and tables are not prescribed, although the various table text styles are provided. The formatter will need to create these components, incorporating the applicable criteria that follow.

ENHANCED PARTICLE SWARM OPTIMIZATION

An enhanced variation and extension to the standard PSO is proposed to produce an intelligent night vision approach. This variation is called an Enhanced PSO [9, 10, 11], though; EPSO has more parameters than the standard PSO. EPSO is utilizing the genetic algorithm procedures, and clearly disagreeing from the standard PSO in that it uses the genetic operators (such as selection, crossover and replacement operations), but yet the numbers of the entire population are still maintained throughout the searching process. The objective (distance) function is handled as a fitness function to guide the searching procedure, making it an easy task to deal with the supposed optimization problem. Still the computation time and the processing power of EPSO do not vary too much from standard PSO’s computation time and processing power.

To evolve the genetic system in the EPSO, we applied the selection, the crossover and the replacement operations as explained below:

A. EPSO Genetic Operators

Selection Operation: The selection algorithm has been proposed to insure a steady convergent behavior of PSO, thus it does not consider all the individuals for the selection algorithm, but only the twenty fittest particles, so, and the EPSO selection procedure is implemented as follows:

The EPSO’s operators are applied to the initial population to get another generation, P (1), and so on.

The fitness function is evaluated for each individual,

providing fitness values, which are then normalized.

The population is sorted by descending fitness values.

The employed procedure in the selection process is focused to find the 20 fittest particles, such that the new population is better than the old population in terms of its fitness.

This procedure is repeated until there are enough selected individuals.

The best individuals in the generation are retained unchanged in the next generation (this process is called elitism or elitist selection).

Arithmetic Crossover: The arithmetic crossover with crossing probability Pc is directed to the selected particles to obtain other best particles. It takes two particles P1 and P2, and performs an interpolation along the line formed by the two parents. By using the arithmetic crossover, we can increase the local diversity of the population.

Replacement Operation: The worst particles in the generated population are replaced by the best particles that are obtained as the result of the crossover operation. Repeatedly replacing the worst particles by the best particles in terms of its fitness can secure that the new population is better than the old population in terms of its fitness.

The goal behind using the genetic system in the PSO system is to control the particles that fly out of the desired domain, or have less fitness values. As a result, these particles can return back to the domain, but with higher fitness values. The implementation of the EPSO can be summarized by the successive steps as shown in the following pseudo-code.

for all particles {

initialize the positions of the particles. An ith particle position is represented as: Pi = (Pi1, Pi2, Pi3, ... PiN).

initialize the particle velocities randomly in the range

, .

and limits in the jth dimension are computed as follows:

Then the constraint on the velocity is computed as follows:

V = v1 + (v2− v1) ∗ rand (N, I); (1)

if V > Vmax

V = Vmax;

endif

if V < Vmin

V = Vmin;

endif

Where R is the chosen number of intervals in the jth dimension, v1 and v2 are two identified constants.

end for }

Evaluate the objective values of the particles (Pbest).

while (the stopping criterion is not reached) {

for all particles {

set Pbest value of the particles as the best positions.

set gbest as global best position among all particles.

end for }

for all particles {

* Calculate new velocities using Equation 1 with the above constraint.

* Apply the selection mechanism as described above.

* Apply the arithmetic crossover (a is a picked random amount) as follows:

c1=P1∗a+P2∗(1−a);

c2=P1∗(1−a)+P2∗a;

* Apply the replacement strategy as described above.

end for }

* Calculate the objective values for the new positions.

If the new value is better than the previous Pbest, the new value is set to Pbest.

The current gbest among the updated Pbest is determined. The current gbest is compared with the previous gbest, and is updated if it is less than the previous gbest.

end while }

PROBLEM FORMULATION

A dataset of images were created under a variety of illuminations, each set consists of three images; one is taken under the condition of enough illumination, and the others are taken under darkish illumination, with one is more darkish than the other two. These images have color values represented by 24 bits in RGB (Red, Green and Blue) color space. The pixel value in 24 bit RGB color scheme forms the input to the system and the output from the developed system is also a 24 bit RGB pixel value after the brightness compensation system. The main goal of the brightness compensation system is to bring images of little illumination values into more illumination values and the illuminative output image must still preserve the characteristics of the input gloomy image.

Proposed Night Visioning Algorithm

The proposed Brightness Compensation System (BCS) starts by dividing the 24 bit input gloomy image into its three channels; Red, Green, and Blue. Then a local visioning model is applied to each component (Red, Green and Blue) separately. The visioning model is a local-based and has been applied to the gray-level distribution in the neighborhood of each pixel. The transformation function is being applied to each pixel of the three components at location (x, y) as shown in Equations 2, 3 and 4:

(2)

(3)

(4)

Then , and are combined together to form the 24 bits RGB image g(x, y).

R(x, y),(x, y) are the corresponding input and output components in the 8 bits Red image, respectively. G(x, y), (x, y) are the corresponding input and output components in the 8 bits Green image, respectively. B(x, y), (x, y) are the corresponding input and output components in the 8 bits Blue image, respectively. Equations 2, 3 and 4 represent the gray-level intensities of the pixels centered at location (x, y). The mean (m) and standard deviation (σ) features were computed in the local neighborhood for each component in the RGB image:

(m) measures the average intensity as defined in Equation 5.

(5)

Where xi is the value of the pixel i , and n is the total number of pixels.

(σ) measures the pixel deviation from the mean as defined in Equation 6[12].

1/2 (6)

As a matter of fact, (m) and (σ) were computed in the local neighborhood of the sliding windows of size 3×3. As shown in the relations below. The parameters α, β,γ and κ are the respective parameters given by the particle P = (αβγκ).

Where α, β, γ, and κ are the parameters defined over the real positive numbers, and are the same for the three components of the RGB image. Adding the last term may brighten and smooth the effects in the whole image. κ may broad the spectrum of the transformation output range, while β and γ may increase the saturation features of the RGB, also, the mean values of Red, Green and Blue should be increased, and those values add more weight to the brightness features, and it probably produce a more brightly image. Saturation can be defined as:

Saturation = mean(max(RGB)-min(RGB))/255;

The above enhancement has been accomplished based on the following assumptions and explanations: β prevents the NAN values since the denominator do not catch a value of zero, γ ≠ 1 allows only a fraction of the mean to be subtracted from the pixel’s input gray-level, and this may broad the spectrum and the chroma distribution of the transformation output range, and also reduces the difference between the gloomy image and its brightly image in terms of PSNR. Overall, the compensation system based on PSO aims to solve the adaptive visioning problem by finding the best combinations of the four tuned real parameters such that the target image should be more brighten than the original gloomy image, and it will approach to the image that was taken under the daylight condition. Though applying PSO to the optimization problems needs two primarily key steps; representation of the solution and the fitness function. However, representation the solution mainly depends on the nature of the problem being solved, in our case; we have four parameters α, β, γ, and κ which are represented as particles, whereas each particle is represented as a string of four real swarms denoting the fourth dimension. Each particle represents a candidate solution to the optimal color night vision problem. Fitness function on the other hand is an important parameter for PSO execution. Alternatively PSO searches for the best global solution according to an objective criterion, in which it should describe the brightness of the image such that it should have a relatively high detail, and high averaging values of the three components of the 24 bit RGB image. Next, another criterion that is closely related to have a relatively high chroma distribution of the image is the entropy feature, in which the entropy indicates that the processed image could be more brightly than the gloomy one. Also, another important criterion that must be introduced in the objective function for the PSO based adaptive visioning approach is the Peak-Signal-to-Noise-Ratio (PSNR) between the input and output images. In addition, PSO tries to find a trade-off between the average brightness and PSNR. The proposed visioning approach has the following properties:

It deals with local neighborhood data, which is represented by the local mean value and the local standard deviation of the neighbor pixels. The mean and standard deviation have been computed for each neighborhood pixels of a block. The block has a size of 3 pixels in both of horizontal and vertical directions respectively, and then the obtained value is to complete the transformation process.

It’s an automatic approach which means that it has no interaction with the human being.

Carrying out PSO can provide the following points to the optimization vision problem:

The fitness criterion of the optimization problem is proportional to three values; the brightness, the entropy and the (PSNR) between the bright and the gloomy images.

It can speed up the convergence process towards the optimal solution.

It has less computation time and complexity than some other optimization techniques.

Entropy (e) measures the randomness in the image as defined in Equation 7 [12].

(7)

The proposed fitness criterion of the PSO-based visioning problem has been chosen on the basis of the following facts and illustrations:

The brightness of the image, which might give an oversized credit to the color night vision, since it will rise up the averaging values of the Red, Green and Blue components.

The peak signal to noise ratio which indicates that the target illuminative image is almost the same as the brightly image; the image which has been taken under enough illumination.

The entropy function which affects on the target image in a positive way, as a matter of fact, a highly chroma distribution values can be obtained from a high value of the entropy.

As a whole, the fitness function shown in Equation 8 is a good choice for a night vision system:

fit(P)=((e(I(P))2+snr(I(P),f(P))2+ABr(I(P))2))1/2 (8)

Where fit(P) is the proposed fitness function for PSO algorithm. I(P) denotes the original image I with the transformation function applied according to Equation 7. e(I(p)) measures the entropy of the gloomy image I(P). snr(I(P),f(P)) computes the SNR between the original gloomy image and the image under processing. ABr(I(P)) returns the average brightness value of the processing image. We can conclude from Equation 8 that PSO aims to find a solution that maximizes the fitness function (fit(P)) by:

Increasing the brightness of the image.

Increasing the PSNR between the gloomy image and the target image.

Increase the overall detail of the processed image, and

Obtain relatively high distributions of the chroma distribution by maximizing the entropic measure.

Whatever the case may be that there should be a balancing between the average brightness and the PSNR values in order no criterion has a more rate than the other, hence there is a restriction on the values of both PSNR and the average brightness.

Average Brightness

The averaged brightness criterion [13] shown in Equation 9 is used to evaluate the algorithms.

(9)

Where Rα (x, y), Gα (x, y), and Bα (x, y) represent the intensity of color components, Red, Green, and Blue, respectively, for the pixel at (x, y) of the image. The average brightness for the input images for some dataset were used in the experiments is shown in Table I. The outputs of the brightness compensation system for the images in our experiments obtained using the proposed approach-based PSO, Differential Evolution (DE) [14, 15, 16] and a conventional Neural Networks with Back Propagation (NNBP) [17, 18] are shown in Table II.

AVERAGED BRIGHTNESS FOR IMAGES USED IN THE EXPERIMENTS

Image

Bright

Gloomy 1

Gloomy 2

Kit

152.56

109.12

13.12

Lady

115.6291

34.1449

13.8634

Natural

109.0356

34.6199

5.5981

Fruit

104.3364

54.3201

20.7587

Clock

111.0786

38.2589

11.4540

Dates

107.5196

48.6510

19.6360

Flowers

111.623411

45.0360

8.2318

THE FITNESS VALUE FOR EPSO, DE AND NNBP

Method

EPSO

DE

NNBP

Image

Gloomy 1

Gloomy 2

Gloomy 1

Gloomy 2

Gloomy 1

Kit

132.16

71.02

128.13

69.100

112.24

Lady

132.1100

85.1329

131.006

79.152

102.47

Natural

101.0126

76.0189

93.146

68.902

87.067

Fruit

104.3364

54.3201

100.346

48.902

90.54

Clock

110.407093

30.809051

99.67

29.89

84.76

Dates

148.934966

56.274217

139.98912

47.120

112.890

Flowers

125.821627

51.196612

112.11

44.113

95.236

Peak Signal to Noise Ratio (PSNR)

The PSNR for images with respect to the target images will be introduced as an evaluation criterion to determine the applicability of the proposed algorithm. The PSNR between two colors images I and is defined as given in Equation 10 [13]:

(10)

(11)

Where R(x, y), G(x, y), and B(x, y) represent the intensities of the components for the image I. (x, y), (x, y), and (x, y) are those for the image . The PSNR values of the compensation system for the images used in our experiments based on using NNBP are shown in Table III.

EXPERIMENTAL SETUP

The control parameters used for running PSO are: The number of particles is 30. Dimension of particles is four since the parameters need to be tuned are four. Range of particles is the positive real numbers. The maximum change one particle can be taken during one iteration is 20. Learning factors or acceleration constants are equal to 1.3. The searching is a repeat process and the stop condition or the maximum number of iterations the PSO executes is set to 100. Inertia weight is set at 0.6 and 0.9. The search space for the parameters were selected in the range, the type of crossover for the PSO is:

The combinations of control parameters used for running DE are: The domain of the parameters was selected in the range:

a = [0, 3.5], b = [0,11], c = [0, 1], and k = [0,10].

The selection mechanism is binary tournament and K-elitism with K = 5. Number of iterations is 200. Population size, crossover probability, and mutation probability are 1000, 0.9, 0.03, respectively. The Experiments are run on MATLAB R2006 using PC with 1.60 GHZ CPU, and 256 MB RAM.

RESULTS AND DISCUSSIONS

An example of three input images from the scene ’Kit’ are shown in Figure 1 (a), (b), and (c). Figure 1 (a) is taken under moderate illumination or daylight condition, whilst the image labeled Gloomy 1 in Figure 1 (b) is taken under the condition of darkish illumination, the image labeled Gloomy 2 in Figure 1 (c) is taken under the condition of more darkish illumination than Gloomy 1.

Figure. 1. The images from the scene ’Kit’ in three illuminations; (a) Bright image, (b) Gloomy 1 image, (c) Gloomy 2 image

A datasets of 32 sets of images have been obtained under the above constraints; each set consists of three images of different brightness and illumination, where one image is taken under daylight condition for the purpose of comparison and another image is almost unclear since it was taken under darkest condition. However, the exact outcomes that are expected to be achieved from this study should contain reconstructing color information from gloomy images and video scenes, also; these results could be applied to real applications such as, mobile robots and cognitive robotics. The output from the system is brightness compensated image. The aim of a brightness compensation system is to bring images of little brightness values into more illumination values.

PSO and DE have been considered as optimization algorithms to solve the night vision problem, in order to evaluate the PSO-based vision method; we compared it with DE-based vision method using the three set images. They are the kit, fruits, vegetables, and chart. The objective evaluation criterion in terms of the fitness score is employed to evaluate the proposed approach; the results for typical runs are given in Table II. The results showed in Table II reveal the following fact: The fitness value using PSO is more when compared with the fitness value using DE and NNBP for the same number of generations. Also it is clear from Table III that PSO-based vision method achieves the highest brightness in the processed gloomy images when compared with the brightness of the gloomy images using DE. This ensures that enhanced PSO method yields better brightness of the gloomy images compared to DE and NNBP. Thus, the above facts reveal the superior properties of PSO when compared with DE and NNBP for the night vision images application.

PSNR FOR IMAGES USED IN THE EXPERIMENTS

Method

EPSO

DE

NNBP

Image

Gloomy 1

Gloomy 2

Gloomy 1

Gloomy 2

Gloomy 1

Kit

13.06

11.12

12.83

10.10

10.24

Lady

14.11

12.19

13.98

12.01

12.87

Natural

14.26

13.97

13.88

11.90

11.06

Fruit

13.33

12.01

13.13

11.97

11.54

Clock

15.93

13.80

14.87

12.81

12.70

Dates

13.05

13.00

12.12

12.10

10.90

Flowers

15.08

14.96

14.41

14.03

12.06

CONCLUSIONS AND FUTURE WORK

In this paper, a new approach to automatic adaptive night vision using real-coded PSO is implemented; the proposed approach is based on specifying a suitable fitness that is proportional to the average brightness, entropy and the PSNR between the target and input images. The objective of the algorithm is to maximize the brightness as well as increasing the entropy and PSNR, thus being able to visualize more color reconstruction in the gloomy images. The algorithm is tested on 32 sets; each set has three images with different illuminations. The results are tabulated and compared with the DE-based the same approach. Also, neural networks with back propagation (NNBP) have been used to reconstruct images from night vision scheme as described in the text. The obtained results indicate that PSO and DE can accomplish the night vision task with good quality and less computational efficiency. For future works, enhanced PSO based night vision method may be extended in several ways, such as: fine tuning of the PSO parameters in order to reduce the number of particles and the maximum number of iterations, another extension is to code local parameters of the method that may be applied to each neighborhood. A third work will focus on different parameter’s values for each component of the 24 RGB image.