Neural Networks

• As old as 1943: Perceptrons. Let us consider the problem solved by perceptron.
• Suppose we have a set of objects that are divided into two classes 0 and 1.
• If there exists a hyperplane that separates the objects 0 and 1, then perceptron can find it as follows:
• arbitrarily choose a vector (hyperplane) and check to see if the hyperplane classifies the objects properly.
• If it doesn't then adjust the equation of the hyperplane as follows
• Wnew = Wold + alpha*(desired-network)*input
• where Wnew is the modified hyperplane, Wold is the old hyperplane. desired is the desired output and network is the network output. input is the input vector and alpha is a system parameter that determines the rate of learning in the range (0,1)
• this algorithm does converge
• Problems in practice
• There are several hyper-planes that will give you the correct solution for the sample space
• The solution for the universe may be different. This problem is not unique to neural networks.
• Most of the time we deal with a problem that is
  • not linear separable
  • not a simple classification into two classes
• That was the major attack on perceptrons by Minsky and Papert. Wrote a book in 1969 called perceptron and identified a whole bunch of problems that couldn't be solved using perceptron.
• linearity was a real limitation for perceptrons to be useful
• For almost 15 years, there wasn't any research
• If we add a layer between input and output we get non-linear equation
• Multilayer feedforward backpropagation network for supervised learning
  • Multilayer: atleast one hidden layer between input and output layer
  • feedforward our calculations of output depend only on the values that were forwarded from the previous layer
    • input layer forwards information to the hidden layer, hidden layer forwards information to the output layer
    • There is no passing of information within a layer.
  • Backpropagation
  • The error is backpropagated from output layer to the hidden layer and from hidden layer to the input layer
    • The error is used to adjust the weights of the connections
  • Supervised learning: For the sample set we know the value of the output
• Design and implementation of a neural network
  • Identify the input and output vectors
  • That is we know the number of neurons in the input layer as well as the output layer
• We collected one days worth of traffic volume data in July. We want to know how much is the average daily traffic volume for the year.
• Input vector has 24 variables
• Output vector is a single variable
• Hidden layer can be average of the input and output vectors
  • rule of thumb
  • a theorem says that if the number of hidden layer neurons is ..... then it can emulate any continuous function
    • You only need one hidden layer
  • roughly 12 hidden layer neurons
• We are going to use SNNS package to run a couple of neural networks
• The file orig.net is the one which has original network.
• Read the file and figure out what part of the file specifies the neurons and what part specifies the connections
• Train the original network look at the results
• Destroy some of the links and compare the results with the original training
• How do we train
  • we need a training set
  • the training set consists of input vectors and their corresponding output value
  • look at orig.pat
  • We have 973 observations: Each observation consists of 24 input values and the corresponding output
  • We want to find a function that will mimic the output value.
• snnsbat orig.cfg orig.log
• the above command runs the snns in batch mode that uses information from the cfg file and logs the training process in the log file
• cfg file specifies the original network file, pattern file, where the trained the network should go and where the results of the trained network should go
• log file gives you details of how the errors reduced as the training progressed.

Other types of networks

• Networks that are not necessarily feedforward
  • Time series analysis
  • what happens on a particular day is dependent on what happened in previous few days
  • The input neurons may send output to the hidden layer and another input neuron on the right hand side to be used for the next time step
  • hidden layer neurons may feedback their output as the input to be used for the next time step
  • that is information may not always flow in forward direction
  • Recurrent, time delay neural networks are some of the examples of these types of networks
• Neo-fuzzy neural networks
  • Only input neurons can be neofuzzy
  • The crisp input is divided into fuzzy segments with a membership value
• 68 F is a crisp value which divided into four fuzzy segments (cold,0), (cool,0.2),(justright,0.7),(warm,0)
• These memberships for the fuzzy segments are really the input to the neural network. So we are taking single crisp value and blowing it up into 4 inputs to the neural network
• see the picture from the blackboard
• There are a lot of different neural network architectures, within an architecture we can change number of neurons, number of connections, the values of the parameters to get different results. So we can never really say that no neural network could solve the given problem. The network that we tried didn't work for the application.

Unsupervised neural networks

• We don't know what class a particular pattern falls in
• But we know that the objects can definitely be grouped into let us say five different classes
• We want to group objects based on 12 input variables into five classes
  • 12 input neurons and 5 output neurons
  • For each output neuron, there is a weight vector.
  • The input vector belongs to the class whose weight vector is most similar to the input vector. All the other output neurons get output value of zero the winner has output value of 1. The weight vector of the winner is modified slightly using the input vector.
  • This is called competitive learning.