Neural Networks: The Forward Pass

Let's start with a fact that is somewhat calming. A neural network is just a mathematical function which we will denote as \(f\). Even though a function, in our code, we shall implement a Python class²,

import numpy as np

class Network:
  def __init__(self, X, y, structure, epochs=20, bt_size=32, eta=0.3):
    pass

For now, you can ignore the input variables of the __init__ definition. We will talk more about them as they become relevant. Before we start thinking about this function \(f\), we need to think about how to model our problems mathematically.

Say the problem is to figure out a way of calculating the area of a square, given its side-length. This problem can be modeled with two numbers: the side-length \(x \in \mathbb{R}\) and an area \(f(x) = y \in \mathbb{R}\). Now we can write the domain and codomain like this \(f:\mathbb{R}\rightarrow \mathbb{R}\).

We have modelled our problem and are aware of what the input and output of \(f\) mean. Say I show you a few example side-lengths and area pairs \((x, y)\), e. g.,

\[ \{(0,0)(1,1)(2,4)(3,9)(4,16)(5,25), ...\} \]

Can you guess the function? Indeed! it's the polynomial \(f(x) = y = x^2\) and you just did a little bit of machine learning in your head. You looked at correct example inputs and outputs and generalised the idea into a general enough function.

In the above problem, we could model the input and output of our problem with just one number. What if instead of a singular value, our problem is best modelled when the inputs and outputs are lists of ordered values: \((\vec{x}, \vec{y})\) vectors? Then the domain and codomain of \(f\) can be written as \(f:\mathbb{R}^n\rightarrow \mathbb{R}^m\).

A subset with \(N\) elements of this domain and codomain or input and output are passed to the Network definition above as the variables X, y. This is our sampled data of known examples. X \(\subset \mathbb{R}^n\) of dimensions/shape \((N, n)\) will be a 2D Numpy array of examples (inputs modelled as row arrays/vectors), where y \(\subset \mathbb{R}^m\) will a Numpy array containing the correct outputs which are also called labels.

Now that we understand the input and output of \(f\), the question is that for some \(\vec{x}\), how should \(f\) map it to the desired \(\vec{y}\)? This process uses a layered architecture with a set of weights \(\mathcal{W}\) and biases \(\mathbf{b}\) and we call it forward-feeding or the forward-pass. We'll learn about forward feeding, weights and biases with a small running example before we generalise. Don't let figure 2 intimidate you. We'll break it down.

For now ignore all the edges and labels and just look at the green, blue and red vertices. If without any further explanation I ask you to tell me how many layers are in this network, you might say three. Then, if I ask you to give me the number of neurons in each layer, you might say \([5, 4, 2]\). You'd be right in both cases!

The first layer in this three-layered Multi-layer Perceptron is the first green column on the left hand side. This layer corresponds to the length of our example input \(\vec{x} \in \mathbb{R}^5\). After the input layer, we have hidden layers. of which in the figure 2's network, there is only one: the blue one with four neurons. Note that even though we have only one hidden layer, it is entirely possible for some other network to have more! After the hidden layers, we'll see the output layer. This layer corresponds to the output \(\vec{y}\). Here we read what the output of our network is after a success forward pass. Thus, the network in figure 2 can be written in the function notation like this, \(f:\mathbb{R}^5\rightarrow \mathbb{R}^2\).

Now that we understand the vertices/neurons in the layers of a network. We are ready to see how the network \(f\) takes \(\vec{x}\) and feeds it forward through all the layers \(l < L\), arriving at the output layer \(L\). The heart of it all is in matrix multiplication. If you don't recall the basics of it, this is a good time to brush-up. The set \(\mathcal{W}\) is a set of matrices; similarly, the set \(\mathbf{b}\) is a set of vectors. For a network with \(L\) layers, we have \(L-1\) many matrices in \(\mathcal{W}\) and vectors in \(\mathbf{b}\).

We know that a simple neural network \(f:\mathbb{R}^n \rightarrow \mathbb{R}^m\) starts with \(\vec{a}^{(1)} = \vec{x} \in \mathbb{R}^n\) then performs \(L-1\) matrix multiplications as shown in equation (1) and (2) and arrives at \(\vec{a}^{(L)} = \vec{y} \in \mathbb{R}^m\). For a network as small as the one shown in figure 2, we can write out the equations for all of its activations. We will also write out the sets \((\mathcal{W}, \mathbf{b})\) with their correctly shaped elements. Since we have three \(L=3\) layers in the Multi-layer Perceptron of figure 2, we will have \(L-1 = 3-1 = 2\) weight matrices and bias vectors.

Let's generalise and implement forward feeding of any given \(\vec{x}\). We'll write a subroutine with a flag. When the flag is true, the subroutine will return all the activations \(\vec{a}\) and outputs \(\vec{z}\) caused by forward feeding \(\vec{x}\), when false, it'll just return \(\vec{a}^{(L)}\).

def forward_pass(self, example, keep_track=True):
  input_layer = example.flatten()
  # if we only want the output of the network
  if keep_track is False:
    for W, b in zip(self.W, self.b):
      input_layer = self.sigmoid(np.dot(W, input_layer) + b)
    return input_layer
  outputs = np.empty(shape=self.L - 1, dtype=np.object)  # z^(l)
  activations = np.empty(shape=self.L, dtype=np.object)  # a^(l)
  activations[0] = input_layer
  for W, b, l in zip(self.W, self.b, range(self.L - 1)):
    outputs[l] = np.dot(W, input_layer) + b
    activations[l + 1] = self.sigmoid(outputs[l])
    input_layer = activations[l + 1]
  return outputs, activations

Beware of bugs in the above code; I have only proved it correct, not tried it. [5] – Donald E. Knuth

It's time to test the code we have so far and see if it performs as per our assumptions. Let's give our Network a string representation. We'll print out information about each layer in our network. This means: the number of neurons, shape of the associated weight matrix and number of elements in the associated bias vector.

def __repr__(self):
  ret = ''
  for l, W, b in zip(self.structure, self.W, self.b):
    ret += '({}: W{} + b{})\n'.format(l, W.shape, b.shape)
  return ret

def __str__(self):
  return self.__repr__()

Let's build the network in figure 2. We know the number of neurons in each of its layers is 5, 4 and 2. Therefore, we'll let structure = [5, 4, 2]. Even though we won't be training this network just yet, we'll still need to pass some mock examples so we can initialise it. For now we can just put this test code in the same file network.py after the Network class definition so we don't have to figure out imports.

X, y = np.array([[2, 3, 4, 5, 7]]), np.array([1, 1, 0, 1, 1])
net = Network(X, y, structure=[5, 4, 2])
print(net)

Running python3 path/to/network.py gets us,

(5: W(4, 5) + b(4,))
(4: W(2, 4) + b(2,))

This prints out the correct information about the weights and biases! (5: W(4, 5) + b(4,)) says that the first layer with five neurons connects with the second layer with 4 neurons using \(W_{4,5}\) and \(\vec{b}\) with 4 elements. Then, (4: W(2, 4) + b(2,)) means that the second layer with four neurons connects with the third layer with 2 neurons using \(W_{2,4}\) and \(\vec{b}\) with 2 elements. So far so good.

We can further investigate if the one-hot encoding was done properly and if the shapes of each activation vector \(\vec{a}^{(l)}\) is correct.

print(net.y)
outputs, activations = net.forward_pass(np.array([1, 0, 1, 0, 1]))
for a in activations:
  print(a.shape)

Running python3 path/to/network.py gets us,

[[0. 1.]
 [0. 1.]
 [1. 0.]
 [0. 1.]
 [0. 1.]]
(5,)
(4,)
(2,)

Can you tell why this output is correct?

The banner [1] of this article is of Matryoshka Dolls. Why? They resemble a Multi-layer Perceptron in structure [2]. Think of the outermost doll as the input layer. She takes the input vector \(\vec{x}\), performs the forward feeding says her little magic spell to get the next activation and passes it onto the next doll within. When the propagation reaches the doll present at the innermost layer, we uncap all the dolls and ask the innermost doll for the output \(\vec{y}\). What if it's not correct \(\hat{\vec{y}} \neq \vec{y}\)?