The Types Of Learning Techniques And Importance

Published Date: 02 Nov 2017

Abstract- Today the most important use of neural network is in classifying the data. The way in which the neural network performs classification varies based on the application of the neural network. The neural network may be trained by giving some input-output pairs or sometimes the classification is carried out without giving such training pairs. In case of classification where training is performed, the rate at which the neural network is trained is very important, because less training time means faster learning and as a result classification of the actual data can be done in less time.

Keywords—Learning Techniques, Supervised Learning, Un-supervised Learning, Hybrid Learning, Learning rate, Classification, Clustering.

Introduction

Classification is basically a set of activities responsible for deriving a model that categorizes and describes classes of data and concepts, whose sole purpose is to determine and predict the classes of objects who have no label. The model derived can be shown in various forms, such as classification (IF-THEN) rules, decision trees, mathematical formulae, or neural network.

A decision tree has a structure similar to a flow chart, where each node denotes a test on an attribute value, each branch denotes the test outcome, and the leaves of the tree represents classes and class distributions.

A neural network, when used for classification, is typically a collection of neuron-like processing units with weighted connections between the units. Fig 1 shown below is the example of a Feed-Forward Artificial Neural Network. Every neural network has one input and one output unit along with zero or more number of hidden units. The input unit receives the input, the output unit generates the classification result and the hidden unit performs the processing.

Fig 1: A Feed-Forward Neural Network

There are some very important components those make up an artificial neuron. These include weighting factor, summation function, transfer function, error function, error and back-propagated value and learning function. The most important among these is the learning function. Its purpose is to modify the variable connection weights when inputs are given to each processing element based on some neural based algorithm.

There are a large number of neural network algorithms that are implemented for different applications. In each type of algorithms, some type of learning technique is always used. This technique decides whether to train a network a teacher (training set) is required or not. The time required for training a network should be very less so that the actual data is classified in less time.

Further this paper is divided into following sections, section II is about training a neural network. Section III is about importance of learning rate in neural network algorithms and section IV is applications of the types of learning technique and section V is conclusion.

Training a neural network

Process of Classification

Every classification technique basically consists of two phases: the learning or the training phase and the other is the output or the testing phase. In the learning phase, the training data whose data objects have a known class label are given as input to the model so as to train the model. Once the model has been trained, the original data set that is to be classified is given as input to the model. The model based on information learned from the training data classifies the test data into appropriate class labels.

Types of Learning

There are basically two types of learning methods that are used. These are: 1) Supervised Learning and 2) Un-supervised Learning.

Supervised Learning

It is a machine learning task of finding a function from training data which is labelled. The training data contains pairs of training examples. In supervised learning, each example is basically a pair consisting of an input data and a desired output data. A supervised learning algorithm goes through the training data and produces an analysed function, which is known as a classifier (for discrete output) or a regression function (for continuous output). The generated function should correctly classify the input object. This can only be done if the learning algorithm generalizes from the training data to unseen situations in a way which is reasonable. Example of supervised learning method is classification.

The vast majority of artificial neural network solutions are trained using the supervised learning technique. In this technique, the actual output generated of a neural network is compared to the desired output. Weights, which are set randomly so that the process can begin, are then adjusted by the network as a result of which in the next iteration, or cycle, it will produce a match that is closer between the desired output and the actual output. This method of learning method tries to reduce the current errors of all elements that do the processing. This global error reduction is created as a result of continuously changing the input weights.

In case of supervised learning, the artificial neural network can only be used once they are trained. Training includes presenting of the input-output data pairs to the network. This data is often known as the training set. That is, for each input set given to the system, the desired output set of the same input should be provided. In almost all applications, actual data must be used. This training phase can consume a large amount of time. Training is considered as complete only when the neural network reaches a performance level defined by the user.

Training sets should be large enough to contain all the information that is needed, if the network has to learn the features and relationships that are important to the network. Not only the sets need to be large but the training sessions should include all the variety of data. Once a supervised network performs well on the training data, then it is important to see what it can do with data it has not seen before. If a system does not give outputs that are reasonable, the training period is not over and training should be continued. Indeed, this testing is the most critical part in supervised learning in order to insure that the network has not simply memorized a given set of data but it has also learned the general patterns that are involved within an application.

Un-Supervised Learning

In machine learning, it refers to the task of trying to find a hidden structure in unlabeled data. Since the examples given to the learner are unlabeled, there is no error or reward signal to evaluate a potential solution. The algorithms or techniques that use un-supervised learning include Clustering and Adaptive Resonance Theory (ART).

This learning method is limited to self-organizing maps. This promising field of unsupervised learning is also known by another name called self-supervised learning. These networks use no external influences to adjust their weights. Instead, they monitor their performance internally. These networks always are in search for regularities or trends in the input signals, and makes changes according to the function of the network. Even without being told whether it's right or wrong, the network still do have some information about how to organize the network on its own.

There are two classes of unsupervised learning: reinforcement and competitive learning. In the first method each input produces a reinforcement of the network weights in such a way as to enhance the reproduction of the desired output. Hebbian learning is an example of a reinforcement rule that can be applied in this case. In competitive learning, the elements of the network compete with each other for the "right" to provide the output associated with an input vector. Only one element is allowed to answer the query and this element simultaneously inhibits all other competitors.

An unsupervised learning algorithm might require the cooperation of clusters of processing elements. In such cases, the clusters would work together to generate results. The cluster's activity as a whole could be increased if some external input activated any node in the cluster. Likewise, if external input to nodes in the cluster was decreased, that could result in an inhibitory effect on the cluster as a whole.

At present, unsupervised learning is not that well understood as supervised learning and so it is still the subject of research. The government has interest in this research because in military they often have situations where they do not have data sets that can be used for training a network until some type of conflict arises.

Importance of learning rate in neural network algorithms

The rate at which ANNs learn depends upon several factors that can be controlled. In selecting the approach there are many trade-offs that can be considered. Obviously, a slower rate means a lot more time is spent in accomplishing the off-line learning to produce a system that is trained adequately. With the faster learning rates, however, the network may not be able to make the fine discriminations that are possible with systems that learn slowly.

Several factors besides time should also be considered when discussing the training task, which is often considered as "tiresome." Network complexity, architecture, type of learning rule or rules employed, size, paradigm selection and desired accuracy must all be considered. These factors play a very important role in determining how long it will take to train a network. Changing any one of these factors may either result in the extension of the training time to an unreasonable length or may even result in an accuracy that is unacceptable.

Most learning functions have some range fixed for a learning rate, or learning constant. Usually this term is considered to be positive and between zero and one. If the learning rate is greater than one, it becomes easy for the learning algorithm to overshoot in adjusting the weights, and the network will start oscillating. Small values of the learning rate will lead to correction that will take place very slowly, but if small steps are taken in correcting errors, then there is a good probability of arriving at the best minimum convergence.

There are many learning laws that are commonly used. Most of these laws are formed based on some sort of variation of the best known and oldest learning law, Hebb's Rule. A few of the very commonly used laws are presented as examples.

Hebb's Rule: It was the first rule developed by Donald Hebb and undoubtedly the best known. His basic rule is: If one neuron receives an input from another similar neuron (mathematically have the same sign) and if both are highly active, the weight between the neurons should be strengthened.

Hopfield Law: It is similar to the Hebb's rule with the exemption that it specifies the degree of the strengthening or the weakening. It states, "Increment the connection weight by the learning rate or decrement the weight by the learning rate, if the desired output and the input are either active or inactive".

The Delta Rule: This rule is a further deviation of Hebb's Rule. It is one of the most commonly used. This rule is based on the simple idea of reducing the difference (the change) between the desired output value and the actual output of a processing element by continuously modifying the strengths of the input connections. This rule changes the weights such that it minimizes the mean squared error of the network. This rule is also known by the name the Widrow-Hoff Learning Rule and the Least Mean Square (LMS) Learning Rule.

The Gradient Descent Rule: In the derivative of the transfer function is still used to modify the delta error before it is applied to the connection weights, which is the similar strategy to that of the Delta Rule. Here, however, an additional proportional constant tied to the learning rate is appended to the final modifying factor acting upon the weight. This rule is commonly used, even though it converges to a point of stability very slowly. It has been shown that different learning rates for different layers of a network help the learning process converge faster.

Kohonen's Learning Law: This procedure was developed by Teuvo Kohonen, who was very much inspired by learning in biological systems. In this procedure, the processing elements compete to get the opportunity to learn, or adjust their weights so as to decrease the training time. The processing element that has the largest output is declared the winner and also it has the capability of inhibiting its competitors and exciting its neighbours. Only the winner plus its neighbours are allowed to adjust their connection weights and only the winner is permitted an output.

Applications

Applications of Supervised Learning

Supervised Learning is a very widely used learning technique. Supervised learning is used when we have a set of training data. This technique is used by back propagation or genetic algorithms. Back propagation uses the target values to calculate the mean square error of the artificial neural network and genetic algorithms use target values when calculating the fitness levels of an individual in a population.

The other applications of supervised learning other than neural networks include speech recognition, deciding whether or not to give credit to a customer, detecting credit card faults, signature detection, deciding whether to buy or sell a stock option and playing games etc.

Applications of Un-Supervised Learning

Un-supervised learning is one in which network parameters are determined as a result of a self-organizing process. Un-supervised learning is a developing field which has applications in fields of clustering, generalization, Hopfield networks, self-organizing maps and travelling salesman problem.

Conclusion

NNs using supervised learning are a ubiquitous tool in machine learning. Any respectable AI expert understands them and their proper domains of application. No ANN course is complete without at least a cursory discussion of the back propagation algorithm, which is fundamental to the field and a great boon for engineering problem solving (involving the discovery of functional mappings).

Also, the very concept of supervised learning in neural systems serves as a gold standard for tutoring, to which all other neural systems can be juxtaposed and thereby assessed for adaptive potential, since, clearly, any system that receives feedback of a supervisory nature has an exceptional starting point for mastering a given problem domain.

Unsupervised learning is the most basic form of adaptivity in neural networks, and clearly a mechanism that operates in many areas of the brain, including the hippocampus and neocortex. Whether cooperative or competitive in nature, the process relies heavily on a fundamental feature of neural systems.

A learning rate is a constant having a value between 0.0 and 1.0. Back propagation is an algorithm that learns using a gradient descent method for searching a set of weights that fits the training data due to which the distance between the network’s class prediction and the known target value of the tuples can be minimized. The learning rate helps getting stuck to a local minimum in decision space and also helps in finding the global minimum. If the learning rate is considered to be too small, then the pace at which learning will be done will be very slow. If the learning rate is considered to be too large, the oscillation between inadequate solutions may occur. A thumb rule is to set the learning rate to 1/t, where t denotes the number of iterations through the training set so far.

ACKNOWLEDGMENTS

Advait Bhatt wishes to thank Asst. Prof. H.B.Jethva, for his guidance and help for doing this work. He also acknowledges Prof. Asst. Prof. D. A. Parikh, Head of computer department, and to all staff of computer department for full support for completion of this work.

Harikrishna Jethva wishes to acknowledge his family and staff of computer department at L.D.College of engineering.