An introduction to neural networks


According to a simplified account, the human brain consists of about ten billion neurons — and a neuron is, on average, connected to several thousand other neurons. By way of these connections, neurons both send and receive varying quantities of energy. One very important feature of neurons is that they don’t react immediately to the reception of energy. Instead, they sum their received energies, and they send their own quantities of energy to other neurons only when this sum has reached a certain critical threshold. The brain learns by adjusting the number and strength of these connections. Even though this picture is a simplification of the biological facts, it is sufficiently powerful to serve as a model for the neural net.

Threshold logic units (TLUs)

The first step toward understanding neural nets is to abstract from the biological neuron, and to focus on its character as a threshold logic unit (TLU). A TLU is an object that inputs an array of weighted quantities, sums them, and if this sum meets or surpasses some threshold, outputs a quantity. Let’s label these features. First, there are the inputs and their weights: X 1,X 2, …, X n and W 1, W 2, …,W n. Then, there are the X i*W i that are summed, which yields the activation level a, in other words:


The threshold is called theta. Lastly, there is the output: y. When a >= theta, y=1, else y=0. Notice that the output doesn’t need to be discontinuous, since it could also be determined by a squashing function, s (or sigma), whose argument is a, and whose value is between 0 and 1. Then, y=s(a).

Threshhold Logic Unit


This TLU has a geometric interpretation that clarifies what is happening. Its four possible inputs correspond to four points on a Cartesian graph. From X 1*W 1+ X 2*W 2 = theta, in other words, the point at which the TLU switches its classificatory behavior, it follows that X 2 = -X 1 + 1.5. The graph of this equation cuts the four possible inputs into two spaces that correspond to the TLU’s classifications. This is an instance of a more general principle about TLUs. In the case of a TLU with an arbitrary number of inputs, N, the set of possible inputs corresponds to a set of points in N-dimensional space. If these points can be cut by a hyperplane — in other words, an N-dimensional geometric figure corresponding to the line in the above example — then there is a set of weights and a threshold that define a TLU whose classifications match this cut.

How a TLU learns

Since TLUs can classify, they know stuff. Neural nets are also supposed to learn. Their learning mechanism is modeled on the brain’s adjustments of its neural connections. A TLU learns by changing its weights and threshold. Actually, the weight-threshold distinction is somewhat arbitrary from a mathematical point of view. Recall that the critical point at which a TLU outputs 1 instead of 0 is when the SUM(X i*W i) >= theta. This is equivalent to saying that the critical point is when the SUM(X i * W i)+ (-1 theta) >= 0. So, it is possible to treat -1 as a constant input whose weight, theta, is adjusted in learning, or, to use the technical term, training. In this case, y=1 when SUM(X i \ W i)+ (-1 * theta) >= 0, else y=0.

During training, a neural net inputs:

  1. A series of examples of the items to be classified
  2. Their proper classifications or targets

Such input can be viewed as a vector: <x 1, X 2, …,X n, theta, t>, where t is the target or true classification. The neural net uses these to modify its weights, and it aims to match its classifications with the targets in the training set. More precisely, this is supervised training, as opposed to unsupervised training. The former is based on examples accompanied by targets, whereas the latter is based on statistical analysis (see resources on the right side for more information). Weight modification follows a learning rule. One idealized learning algorithm goes something like this:

Listing 1. Idealized learning algorithm
fully_trained = FALSE
DO UNTIL (fully_trained):
    fully_trained = TRUE
            #Weights compared to theta
        a = (X1  W1)+(X2  W2)+...+(Xn * Wn) - theta
        y = sigma(a)
        IF y != target:
            fully_trained = FALSE
        FOR EACH Wi:
        MODIFY_WEIGHT(Wi)          #According to the training rule
    IF (fully_trained):

You’re probably wondering, “What training rule?” There are many, but one plausible rule is based on the idea that weight and threshold modification should be determined by a fraction of (t – y). This is accomplished by introducing alpha (0 < alpha < 1), which is called the learning rate. The change in W i equals (alpha (t – y) Xi). When alpha is close to 0, the neural net will engage in more conservative weight modifications, and when it is close to 1, it will make more radical weight modifications. A neural net that uses this rule is known as a perceptron, and this rule is called the perceptron learning rule. One result about perceptrons, due to Rosenblatt, 1962 (see resources on the right side for more information), is that if a set of points in N-space is cut by a hyperplane, then the application of the perceptron training algorithm will eventually result in a weight distribution that defines a TLU whose hyperplane makes the wanted cut. Of course, to recall Keynes, eventually, we’re all dead, but more than computational time is at stake, since we may want our neural net to make more than one cut in the space of possible inputs.

Our initial puzzle illustrates this. Suppose that you were given N characters of code that you knew to be either C, C++, Java, or Python. The puzzle is to construct a program that identifies the code’s language. A TLU that could do this would need to make more than one cut in the space of possible inputs. It would need to cut this space into four regions, one for each language. This would be accomplished by training a neural net to make two cuts. One cut would divide the C/C++ from the Java/Python, and the other cut would divide the C/Java from the C++/Python. A net that could make these cuts could also identify the language of a source code sample. However, this requires our net to have a different structure. Before we describe this difference, let’s briefly turn to practical considerations.

Figure 2. Preliminary (and insufficient) Perceptron Learning Model
Preliminary (and insufficient) Perceptron Learning Model



Table 1. Intermediate nodes mapped to output (as Boolean functions)
N1 N2 O1 (C) O2 (C ) O3 (Java) O4 (Python)
0 0 0 0 0 1
0 1 0 0 1 0
1 0 0 1 0 0
1 1 1 0 0 0


The delta rule


Limiting ourselves to nets with no hidden nodes, but possibly having more than one output node, let p be an element in a training set, and t(p,n) be the corresponding target of output node n. However, let y(p,n) be determined by the squashing function, s, mentioned above, where a(p,n) is n’s activation relative to p, y(p,n) = s( a(p,n) ) or the squashed activation of node n relative to p. Setting the weights (each Wi) for a net also sets the difference between t(p,n) and y(p,n) for every p and n, and this means setting the net’s overall error for every p. Therefore, for any set of weights, there is an average error. However, the delta rule rests on refinements in the notions of average and error. Instead of discussing the minutiae, we shall just say that the error relative to some p and n is: ½ * square( t(p,n) – y(p,n) ) (see resources on the right side for more information). Now, for each Wi, there is an average error defined as:

Listing 2: Finding the average error
sum = 0
        sum = sum + (1/2  (t(p,n)-y(p,n))^2)
average = 1/M  sum



X(p,i,n) is the i th element in p that is input into node n, and alpha is the already noted learning rate. Finally, s'( a(p,n) ) is the rate of change (derivative) of the squashing function at the activation of the n th node relative to p. This is the delta rule, and Widrow and Stearns (see resources on the right side for more information) showed that when alpha is sufficiently small, the weight vector approaches a vector that minimizes error. A delta rule-based algorithm for weight modification looks like this.

Downward slope (follow until error is suitably small)
step 2: for each i, change Wi by:
            alpha  s'(a(p,n))  (t(p,n)-y(p,n)) * X(p,i,n)

There are important differences from the perceptron algorithm. Clearly, there are quite different analyses underlying the weight change formulae. The delta rule algorithm always makes a change in weights, and it is based on activation as opposed to output. Lastly, it isn’t clear how this applies to nets with hidden nodes.



Figure 3. “Code Recognizer” back-propagation neural network
Code Recognizer back-propagation neural network



where d(n) for a hidden node n, turns on (1) how much n influences any given output node; and (2) how much that output node itself influences the overall error of the net. On the one hand, the more n influences an output node, the more n affects the net’s overall error. On the other hand, if the output node influences the overall error less, then n’s influence correspondingly diminishes. Where d(j) is output node j’s contribution to the net’s overall error, and W(n,j) is the influence that n has on j, d(j) * W(n,j) is the combination of these two influences. However, n almost always influences more than one output node, and it may influence every output node. So, d(n) is:







Initialize Wi to small random values.

Steps to follow until error is suitably small

Step 1: Input training vector.
Step 2: Hidden nodes calculate their outputs.


Recognizing success

With the back-propagation algorithm in hand, we can turn to our puzzle of identifying the language of source code samples. To do this, our solution extends Neil Schemenauer’s Python module bpnn (see resources on the right side for more information). Using his module is amazingly simple. We customized the class NNNN in our class NN2NN2, but our changes only modify the presentation and output of the process, nothing algorithmic. The basic code looks like:

Listing 3: Setting up a neural network with

#create the training and testing data
trainpat = []  
testpat = []  
for n in xrange(TRAINSIZE+TESTSIZE):  
            #... add vectors to each set
#train it with some patterns

#test it 

#report trained weights 


Â Â Â python py c java


Listing 4: Code2Data output vectors

Recall that the input values are normalized numbers of occurrences of various special characters. The target values (after the greater than sign) are YES/NO representing the type of source code file containing these characters. But there is nothing very obvious about what is what. That’s the great thing, the data could by _anything_that you can generate input and targets for.


Listing 5: Running
> py c java | > test_results.txt 
Total bytes of py-source: 457729 
Total bytes of c-source: 245197 
Total bytes of java-source: 709858 
Input set: ) (  . = ; " , ' * / { } : - 0 + 1 [ ] 
INPUTS = 20 
0.086... 0.085... 0.084... 
Success rate against test data: 92.60%

The decreasing error is a nice reminder, and acts as a sort of progress meter during long runs. But, the final result is what is impressive. The net does a quite respectable job, in our opinion, of recognizing code — we would love to hear how it does on your data vectors.


We have explained the basics of the neural net in a way that should allow you to begin to apply them in your own coding. We encourage you to take what you have learned here and attempt to write your own solution to our puzzle. (see resources on the right side for our solution.)

Andrew Blais
David Mertz