The Perceptron and Friends

Lesson 2

Introduction

We've seen in our previous lesson that we can simulate a neural cell at the very low level of its electro-chemical processes. Indeed, this is quite helpful when trying to understand the functioning of individual cells and their transmission of the neural impulse (i.e. the progression of the action potential).

Our interests for this course lie mainly in the information processing actions of several (sometimes many many more) interconnected neurons. Computationally we can just ignore the process that causes the neuron to fire and, instead, concentrate on the net result of that firing. Should you protest our simplification, say, along the lines of  “Well, couldn't the low level effects of the electro-chemical process have some slight effect on our result?” I present you with the following observation— Simulation and modelling must take place at some scale of analysis. Failure to fix our scale or scope predestines us to fail miserably. Should we model the electro-chemical processes? We're close to modelling some of the molecular processes when we do this, but what about the atomic processes themselves? What of the sub-atomic processes? Will our results hold up in different physical universes, those where some of our closely held rules don't apply?

I  hope you appreciate that we're mainly interested in the net result of the electro-chemical and synaptic processes– that is, the firing of neurons and their connections to their neighbors, and how those firings can communicate information.

The Perceptron

In the late 50's - early 60's Frank Rosenblat proposed several machines that laid the groundwork for the advent of artificial neural networks. These machines were called Perceptrons. These machines were based on the 1940's work of Warren McCulloch and Walter Pitts on the logical calculus of neural mechanisms. [1,2,3]

Rosenblat's machines were collections of units like those shown below. For this reason, the name perceptron has been used to denote not only the machines but (more commonly) the processing units themselves. Variations of these units go by the name Adaline, Sigma-node, and others.

Figure 1. A basic perceptron. Inputs, , are multiplied by their respective weights, . These weighted inputs are summed and, if the resulting value exceeds some threshold, θ, the output Ψ is a spike, usually represented by a ‘1’. Otherwise, there is no output, a ‘0’.

The perceptron is an analog to the biological neurons we described in the previous section.  In essence, the perceptron is an artificial neuron that processes information from several inputs and, using some decision rule, and either activates its output or not.The inputs are analogous to the dendrites and the output represents the axon in a biological neuron. A collection of these assemblies will connect the outputs (axons) of some cells to the inputs of others, creating a virtual, artificial assembly.

In short, the perceptron collects several inputs, weights them, sums them, and then makes a decision to ‘fire’ or not based on this weighted sum.

Typically the inputs are binary – that is,they either are 1 or 0, representing the presence or absence of activity respectively. In fact, biological neurons behave largely in this manner. There is no differentiation between ‘strengths’ of firing, only the presence or absence thereof. Since the output of a perceptron is binary it follows that the resulting input to other cells will also be binary. The weights act to condition these inputs, emphasizing some and de-emphasizing others before they are summed. After this summation a decision is made to fire or not. The most basic type of decision is a so-called threshold decision. We will examine various sorts of decision functions in the upcoming sections.

Notation

There are a many ways to notate the various inputs, weights, and other parameters, Here, we use the Greek lowercase psi, ψ, to denote the input n-tuple. The subscript selects one of the elements from the list, so is the 3rd input. (NB- Some folks refer to a list of this sort as a vector. As a point of fact, this a semantic matter of interpretation, most notably with respect to magnitude. Usually the nature of the input of a neuron/perceptron is not interpreted to have a magnitude. Still, the term ‘input vector’ is frequently used and should be understood to be simply the list of inputs to the neuron. Some argue that the general form of these sorts of data structures follow more closely the idea of tensors. You are left to decide for yourself what makes the most sense and I welcome the feedback of any philosophical debates on the subject.)

Similarly, the weights are represented by the Greek lowercase letter alpha, α. Subscripts represent individual elements of the list, being the third weight.

Summation is, as is typical, represented by the Greek uppercase sigma, Σ.

Finally, the input to the threshold function shown above is represented by the lowercase Greek letter theta, θ.

The resulting state of the cell, on or off, is a binary value represented by the capital psi, Ψ.

Sometimes, for convenience, the input and output are represented by lowercase Roman letters 'i' and 'o' respectively with weights represented by 'w'.

Another common notation uses to represent the input, to represent the weights, and to represent the output state of the cell.

Traditional mathematical notation for the above process is

Where θ represents the Heaviside unit step function. In alternate implementations θ is an adjustable threshold value. We will investigate several types of threshold functions (also called activation functions) below.

First, we'll decompose the process for those of you not totally familiar with this sort of calculation.

Simple Summation

To quote the philosopher Mr. T, “enough jibberjabber”, let's get on with some computational examples. Let's start by examining the process of summation.

Figure 2. Examining the inputs and summation.

In Mathematica a list or vector or n-tuple is represented as a comma separated list, surrounded by curly-brackets (also known as braces or curly braces).

Above we see a 5-tuple representing an input. The first and last two inputs are firing neurons, while the second and third are inactive.

We can sum the values of the list using this notation-

Here, Plus is said to be applied to the list (the two '@' signs denote application, shorthand for the Mathematica function.) and we see the result, 3. So, in some overly simplistic, majority rules environment, we might say that, for a 5-input perceptron if 3 or more inputs are firing then we too shall fire.

This may be adequate but it doesn't allow any sort of fine-tuning of the inputs, it doesn't allow us to say that some inputs are more important than others, for example –

(The semicolon at the end of the line suppresses output from Mathematica. ψ is set, we just don't see the results of setting it.)

In this case there are still three inputs firing but we can't differentiate the resulting sum from the previous example.

Explore

   1.   Try other combinations and different sizes of input vectors.

   2.    Can you figure out other ways to represent this process in Mathematica?

Example answers

   1.   Try other combinations and different sizes of input vectors.

   2.    Can you figure out other ways to represent this process in Mathematica?

webMathematica

These exercises are in file L2E1.js (link!)

Weighted Summation

Figure 3. Weighting the inputs and summing them

Here we see our input ψ along with a set of weights α —

Multiplying the input vector by the weight vector

results in a new set of values. This is sometimes called an activation vector. We sum the activation just like before —

This final value (sometimes notated ) is called the activation value or net activation of the perceptron.

In this case, different combinations of inputs will result in different output-

So, you see here that the result of the summation isn't always binary (0,1) but rather takes on some real value as a result of the summation.

Explore

   1.   Try other combinations and different sizes of input vectors.

   2.  Try different weight vectors.

   3.   Can you figure out other ways to represent this process in Mathematica?

Example answers

   1.   Try other combinations of input vectors.

   2.   Try different weight vectors.

   3.    Can you figure out other ways to represent this process in Mathematica?

If students do this directly, sing!

webMathematica

These exercises are in file L2E2.js (link!)

Weighted Summation with Threshold

Figure 4. Adding the threshold function.

In a neural network, both physical and artificial, the outputs of neurons (axons) serve as inputs for other neurons (synapses). You may recall that the decided for this experiment to constrain ourselves to an all-or-nothing perceptron— i.e. either the artificial neuron fires or it does not. We represented these two states numerically by 1 and 0 respectively. We have a net activation value of in this example but we need to convert it to one of the two states above, 0 or 1.

Mathematica's function returns 0 if its input is less than 0 and 1 otherwise. Therefore, we can use it as a simple decision function: If the sum of the weighted inputs is greater than 0 than we ‘fire’ (i.e. we pass a value of 1 down to other perceptron inputs), otherwise, we do not. As we stated above, the UnitStep function is an implementation of the Heaviside unit step function.

Here is the math we're already familiar with —

the weighted inputs —

summed —

and finally passed through the UnitStep function —

Thus, for the given input ψ and weights α our perceptron fires!

Notationally, as seen in Equation 1, we represent the unit step function with θ(x). Mechanically, this is what the function does —

Other neural network notate the step function as a special type of output function (we'll look at different output functions below). In this case the ‘generic’ output function is notated and the specific case of the step function .

Explore

   1.   Try other combinations and different sizes of input vectors, weight vectors.

   2.   Modify the example to have an adjustable threshold, θ.

   3.   Pick a set of random input values and weights. Hand tweak the inputs to get the neuron to fire or not. Then, hand tweak the weights to achieve the same results.

   4.   Can you figure out other ways to represent this process in Mathematica?

Example answers

   1.   Try other combinations and different sizes of input vectors, weight vectors.

   2.  Modify the example to have an adjustable threshold, θ.

Here, the threshold is

   3.   Pick a set of random input values and weights. Hand tweak the inputs to get the neuron to fire or not. Then, hand tweak the weights to achieve the same results.

Generate a random table of inputs

and weights

and carry out the process-

It didn't fire. Tweak the input vector (turn off the two bad values and turn on a positive weighted one)

carry out the process —

It fires!

Reset ψ to its original value, modify one member of α and carry out the process —

   4.    Can you figure out other ways to represent this process in Mathematica?

NB- To write a Mathematica function to do the thresholding for us we could say

This function takes two arguments, the value to threshold and the threshold value.

Evaluate the following and double-click on a graphic to see an animation of the function as θ changes on the range [-1,1].

webMathematica

These exercises are in file L2E3.js (link!)

A Little Shortcut

Recall that, based on Equation 1, the net activation, is

There is a convenient mathematical shortcut for generating the net activation according to the above equation. The does exactly what the above equation specifies - it multiplies two vectors together and sums the resulting values. (Can you think of other mathematical / geometric applications where you've seen this process before? That is, multiplying things together and adding them all up?)

In notation, the dot product is denoted

The dot-product takes two vectors as an input and returns a single value (called a scalar)

Explore

   1.   Try other combinations and different sizes of input vectors, weight vectors.

   2.   What happens if the two input vectors have different lengths? Does this even make sense?

   3.   The dot product has an interesting geometric interpretation. Use Mathematica to explore these.

Example answers

   1.   Try other combinations and different sizes of input vectors, weight vectors.

   2.  What happens if the two input vectors have different lengths? Does this even make sense?

Of course there are unequal amounts of things to multiply together and eventually add. Therefore no.

   3.  The dot product has an interesting geometric interpretation. Use Mathematica to explore these.

to come.

webMathematica

These exercises are in file L2E4.js (link!)

The Output Function

In a neural network, both physical and artificial, the outputs of neurons (axons) serve as inputs for other neurons (synapses). You may recall that the decided for this experiment to constrain ourselves to an all-or-nothing network— either the neuron fired or it did not. We represented these two states numerically by 1 and 0 respectively. We have a net activation value of in this example but we need to convert it to one of the two states above, 0 or 1.

This transformation is achieved using an output function (also sometimes called a transfer function, or activation function). Notationally, we will generically use to indicate the output function.

Step Function

We've already looked at the simple unit step function, θ(x) or , above.

As an equation the step function is simply —

In Mathematica, functions are denoted —

We are using so-called Traditional Notation in Mathematica, so we can get away with a little prettier version-

Recall that the step function looks like this —

Linear Function

One frequently used activation function is the simple linear transfer —

In Mathematica —

and graphically —

Notice that this output function simply replicates its input. In this case notice that we've strayed from our original biologically-correct version of all-or-nothing activation. Still, these sorts of activation functions will have further utility down-the-road.

Theta Function

Confusing the situation a little bit, there is a sometimes used function called that isn't just a unit step, instead it is a modification of the unit step as so –

The result is a linear function, but only when .

In Mathematica —

and graphically —

Sigmoid Function

The sigmoid function is a commonly used activation function in artificial neural networks. Mathematically it is —

and graphically —

Gaussian Function

Another statistically interesting function is the Gaussian activation function —

Graphically —

Explore

   1.   Speculate on the use of each of these functions. What are some of them good at? What are they bad at?

   2.   Speculate on the statistical nature of the last two activation functions (the Sigmoid and Gaussian functions). Why would we want such a thing?

   3.   Take one of the full example from above and apply the different output functions to it (instead of UnitStep[]) does it change the resulting activation value? If so how?

Example answers

   1.   Speculate on the use of each of these functions. What are some of them good at? What are they bad at?

to come

   2.   Speculate on the statistical nature of the last two activation functions (the Sigmoid and Gaussian functions). Why would we want such a thing?

to come

   1.   Take one of the full example from above and apply the different output functions to it (instead of UnitStep[]) does it change the resulting activation value? If so how?

webMathematica

These exercises are in file L2E5.js (link!)

References

   [1]   McCulloch, WS, Pitts, W. 1943, 'A logical calculus of the ideas immanent in nervous activity', Bulletin of Mathematical Biophysics, vol. 5, pp 115–133.

   [2]   Rosenblatt, F. 1959, ‘Two theorems of statistical separability in the perceptrton’, Proceedings of a Symposium on the Mechanization of Thought Processes, Her Majesty's Stationary Office, London, pp. 421–456.

   [3]   Rosenblatt, F. 1962, Principles of Neurodynamics, Spartan Books, New York.

Revision History

   1.  October 2003 - Initial ramblings version

   2. December 2003 - First release to CSAC

Created by Mathematica  (December 12, 2003)