The History Of Artificial Neural Networks

Published Date: 02 Nov 2017

D:\download 2013\Downloads\images (20).jpg

Neural Networks have tree-like networks of nerve fibers called dendrites are connected to the cell body or soma, where the cell nucleus is located. Extending from the cell body is a single long fiber called the axon, which eventually branches into strands and sub strands, and are connected to other neurons through synaptic terminals or synapses. The transmission of signals from one neuron to another at synapses is a complex chemical process in which specific transmitter substances are released from the sending end of the junction. The effect is to raise or lower the electrical potential inside the body of the receiving cell. If the potential reaches a threshold, a pulse is sent down the axon and the cell is ‘fired’[2]. The decision making process of the ANN is more holistic, based on the aggregate of entire input patterns, whereas the conventional computer has to wade through the processing of individual data elements to arrive at a conclusion. The neural networks derive their power due to their massively parallel structure, and an ability to learn from experience. They can be used for fairly accurate classification of input data into categories, provided they are previously trained to do so. The accuracy of the classification depends on training, which in turn depends upon the rigor and depth of the training. The knowledge gained by the learning experience is stored in the form of connection weights, which are used to make decisions on fresh input. ANN models can detect patterns that relate input variables to their corresponding outputs in complex biological systems for prediction. Methods for improving network performance include finding an optimum network architecture and appropriate number of training cycles, using different input combinations.

Three issues need to be settled in designing an ANN for a specific application: (i) topology of the network; (ii) training algorithm and (iii) neuron activation function. A network may have several ‘layers’ of neurons and the overall architecture may either be feedback or feed forward structure. Artificial neural networks (ANN) [1] have been developed as generalizations of mathematical models of biological nervous systems. A first wave of interest in neural networks (also known as connectionist models or parallel distributed processing) emerged after the introduction of simplified neurons by McCulloch and Pitts [2].The basic architecture consists of three types of neuron layers: input, hidden, and output layers. In feed-forward networks, the signal flow is from input to output units, strictly in a feed-forward direction. The data processing can extend over multiple (layers of) units, but no feedback connections are present. Recurrent networks contain feedback connections. Contrary to feed-forward networks, the dynamical properties of the network are important.

The computational model of a neuron can be described by the

y=f((1)

equation where y is the output value transferred further by all the output connections, xk is the kth input, wk is the weight of the connection related to the kth input and f is the activation function which is usually the signum (sign) or the sigmoid (¾) function defined as

sign(x) = (x) = ………..(2)

A neuron can be efficiently trained by the perceptron learning algorithm to solve any linearly separable problem.However, most of the problems are not linearly separable,so an individual neuron cannot give an acceptable solution.For these problems the neurons have to be structured and connected to form a network.

A neural network has to be configured such that the application of a set of inputs produces the desired set of outputs [1]. Various methods to set the strengths of the connections exist. One way is to set the weights explicitly, using a priori knowledge. Another way is to train the neural network by feeding it teaching patterns and letting it change its weights according to some learning rule. The learning situations in neural networks may be classified into three distinct sorts. These are supervised learning, unsupervised learning, and reinforcement learning. In supervised learning, an input vector is presented at the inputs together with a set of desired responses, one for each node, at the output layer. A forward pass is done, and the errors or discrepancies between the desired and actual response for each node in the output layer are found. These are then used to determine weight changes in the net according to the prevailing learning rule. The term supervised originates from the fact that the desired signals on individual output nodes are provided by an external teacher. The best-known examples of this technique occur in the back propagation algorithm, the delta rule, and the perceptron rule. In unsupervised learning (or self-organization),a (output) unit is trained to respond to clusters of pattern within the input. In this paradigm, the system is supposed to discover statistically salient features of the input population. Unlike the supervised learning paradigm, there is no a priori set of categories into which the patterns are to be classified; rather, the system must develop its own representation of the input stimuli. Reinforcement learning is learning what to do – how to map situations to actions – so as to maximize a numerical reward signal. The learner is not told which actions to take, as in most forms of machine learning, but instead must discover which actions yield the most reward by trying them. In the most interesting and challenging cases, actions may affect not only the immediate reward, but also the next situation and, through that, all subsequent rewards. These two characteristics, trial-and error search and delayed reward are the two most important distinguishing features of reinforcement learning.

Problem of neural network

The learning process is usually done in epochs, a complete training pass through the entire sample set. The most popular and widely used method is the back propagation algorithm which has many problems leading to an inefficient, very slow learning process. Because back propagation is a gradient descent algorithm, it converges only to a local minimum. A popular way to handle this problem is to initialize the network with random weights and repeat the whole learning process with different randomizations. This approach increases the computational efforts seriously and gives no guarantees to reach the global or even a close local optimum. The back propagation algorithm has a step size (sometimes called learning rate) parameter which determines how much the error is corrigated by the weights in a given step. The step size can be global for the entire network or local for the individual weights. It also can be static for the whole training process or can be adaptively changed during that. Unfortunately it is far from obvious how to select the step size properly. If it is too small then the learning can become very slow. If it is too big, the learning can diverge without reaching a solution[8].

During learning each neuron tries to be a feature detector and so plays a useful role in the network’s behavior. But because most of the neurons do not communicate with each other, they learn independently according to a common error signal computed by the difference between the network’s and the desired outputs. If the error signal is strong enough, a neuron can change its target and decides to be a detector of another feature. Without synchronization the independent parts cannot quickly agree on the feature targets and so they keep changing for a long time. This fact can significantly slow down the learning convergence. It is known as the moving target problem. Another important issue of neural networks is to define their structure which can really affect the learning capabilities of the network. One can use a standard structure as the fully connected feedforward ANN generate it by using an evolution algorithm (which is computationally expensive) or use the cascade-correlation architecture.

Types of Artificial Neural Networks

Feed-Forward Neural Network

Feed forward back propagation artificial neural network model shown in Fig.1 consists of input, hidden and output layers. Back propagation learning algorithm was used for learning these networks. During training this network, calculations were carried out from input layer of network toward output layer, and error values were then propagated to prior layers. Feed forward networks often have one or more hidden layers of sigmoid neurons followed by an output layer of linear neurons. Multiple layers of neurons with nonlinear transfer functions allow the network to learn nonlinear and linear relationships between input and output vectors. The outputs of a network such as between 0 and 1 are produced, then the output layer should use a sigmoid transfer function (tansig)

FB artificial intelligence model consists of input, hidden and output layers. Backpropagation learning algorithm was used for learning these networks. During training this network, calculations were carried out from input layer of network toward output layer, and error values were then propagated to prior layers. Feedforward networks often have one or more hidden layers of sigmoid neurons followed by an output layer of linear neurons. Multiple layers of neurons with nonlinear transfer functions allow the network to learn nonlinear and linear relationships between input and output vectors. The linear output layer lets the network produce values outside the range –1 to +1. On the other hand, outputs of a network such as between 0 and 1 are produced, then the output layer should use a sigmoid transfer function (logsig)

Radial Basis Function (RBF) Network

The Radial Basis Function (RBF) network is a three-layer feed-forward network that uses a linear transfer function for the output units and a nonlinear transfer function (normally the Gaussian) for the hidden layer neurons[6]. Radial basis networks may require more neurons than standard feed-forward back propagation networks, but often they can be designed with lesser time. They perform well when many training data are available. Much of the inspiration for RBF networks has come from traditional statistical pattern classification techniques. The input layer is simply a fan-out layer and does no processing. The second or hidden layer performs a nonlinear mapping from the input space into a (usually) higher dimensional space whose activation function is selected from a class of functions called basis functions. The radial-basis-function (RBF) based neural network is well suited for such cases[4,5,6]. The arbitrariness in the design of a neural network trained by a BP algorithm, for its number of hidden layers and the number of neurons per hidden layer, is removed in the RBF neural network by having only one hidden layer and the number of neurons in the hidden layer optimized for a particular system set of input pattern of data The radial basis function network (RBFN) has a feed forward structure consisting of three layers, an input layer, a nonlinear hidden layer and a linear output layer, as shown in Fig. 1. The hidden nodes are the radial basis function units and the output nodes are simple summations.

D:\download 2013\Downloads\BianconiFig4.png

This particular architecture of RBFN has proved to directly improve training and performance of the network [4].

RBF networks have their origin in the solution of the multivariate interpolation problem [20, 41. These networks have traditionally only one hidden layer. Properly trained, they can approximate an arbitrary function f : ---f R by mapping: m

………………………(i)

where, x E Rn, {wz; z = 1, . . . , m} denotes the weights coefficients, wo is the bias and zj (x) represents the activationfunction (also known as radial basis function)[7]

Any of the functions, namely spline, multiquadratic and Gaussian function, may be used as a transfer function for the hidden neurons. The Gaussian RBF, which is the most widely used one, has been considered for the proposed fault classification and location applications.

Cascade forward back propagation neural networks

CFBP models are similar to feed-forward networks, but include a weight connection from the input to each layer and from each layer to the successive layers. While two-layer feedforward networks can potentially learn virtually any input-output relationship, feed-forward networks with more layers might learn complex relationships more quickly. Cascade forward back propagation ANN model is similar to feed forward back propagation neural network in using the back propagation algorithm for weights updating, but the main symptom of this network is that each layer of neurons related to all previous layer of neurons Tan-sigmoid transfer function was used to reach the optimized status. The three-layer network also has connections from the input to all three layers. The additional connections might improve the speed at which the network learns the desired relationship[4].CF artificial intelligence model is similar to feedforward backpropagation neural network in using the back propagation algorithm for weights updating, but the main symptom of this network is that each layer of neurons related to all previous layer of neurons [5]Tan-sigmoid transfer function, log - sigmoid transfer function and pure linear threshold functions were used to reach the optimized status[4].The performance of cascade forward backpropagation and feedforward backpropagation were evaluated using Root Mean Square Error (RMSE).