Graphing the error landscape for the OR network

The error landscape for the OR network shows how the total error (or TSS) changes as the values of the weights in the network change. To graph the error equation for a network, you need several things:

1. Describe the parts of the network:
   1. The OR network has two inputs and one output unit.
   2. Let the units in the network be i1, i2, and o1.
   3. Let the weights be w1 and w2.
   4. Let the bias be b1.
2. The independent variables (x and y axes) that will be graphed may be varied in the OR network, as there are three independant variables: the weights in the network (w1 and w2) and the bias (b1). The dependent variable is the TSS (or total sum of square error) value for each set of weights. The TSS is calculated from:

   TSS = Σ (target_p - output_p)²

3. In the OR network there are four patterns to consider.

   	i1	i2	o1
   p0	0	0	0
   p1	0	1	1
   p2	1	0	1
   p3	1	1	1

   So the TSS can be calculated as:
   

   TSS = (target_p0 - output_p0)² + (target_p1 - output_p1)² + (target_p2 - output_p2)² + (target_p3 - output_p3)²

   = (0 - output_p0)² + (1 - output_p1)² + (1 - output_p2)² + (1 - output_p3)²

   = output_p0² + (1 - output_p1)² + (1 - output_p2)² + (1 - output_p3)²

4. The output for each pattern is given by the activation of the output unit, o1.
   The equation for the output unit is given by:

   o1 = f(i1*w1 + i2*w2 + b1)

   where f(x) is the sigmoid equation given by:

   f(x) = 1/(1+e^-x)

5. Substituting in for each pattern gives the following:
   For pattern p0:

   o1 = f(0*w1 + 0*w2 + b1) = f(b1)

   For pattern p1:

   o1 = f(0*w1 + 1*w2 + b1) = f(w2+b1)

   For pattern p2:

   o1 = f(1*w1 + 0*w2 + b1) = f(w1+b1)

   For pattern p3:

   o1 = f(1*w1 + 1*w2 + b1) = f(w1+w2+b1)

6. Substituting back into the TSS equation gives the final equation:

   TSS == output_p0² + (1 - output_p1)² + (1 - output_p2)² + (1 - output_p3)²

   = (1/(1+e^-b1))² + (1 - (1/(1+e^-(w2+b1))))² + (1 - (1/(1+e^-(w1+b1))))² + (1 - (1/(1+e^-(w1+w2+b1))))²

7. As previously mentioned, it is possible to create multiple interesting graphs from the three available variables.
   Firstly, we will create a single variable graph by assigning the value 1 to both w1 and w2. This gives us:

   (1/(1+e^-b1))² + 2 * (1-(1/(1+e^-(1+b1))))² + (1-(1/(1+e^-(2+b1))))²

   Transforming this equation into the required form for the GA tutorial, yields:

   (1/(1+e^(-x)))^2 + 2 * (1-(1/(1+e^(-(1+x)))))^2 + (1-(1/(1+e^(-(2+x)))))^2

   Or for use in gnuplot:

   (1/(1+exp(-x)))**2 + 2 * (1-(1/(1+exp(-(1+x)))))**2 + (1-(1/(1+exp(-(2+x)))))**2

8. Secondly, we will create a two variable graph by assigning the same value to w1 and w2. This gives us:

   (1/(1+e^-b1))² + 2 * (1-(1/(1+e^-(w1+b1))))² + (1-(1/(1+e^-(2*w1+b1))))²

   Transforming this equation into the required form for the GA tutorial, yields:

   (1/(1+e^(-x)))^2 + 2 * (1-(1/(1+e^(-(y+x)))))^2 + (1-(1/(1+e^(-(2*y+x)))))^2

   Or for use in gnuplot:

   (1/(1+exp(-x)))**2 + 2 * (1-(1/(1+exp(-(y+x)))))**2 + (1-(1/(1+exp(-(2*y+x)))))**2