Literature
This page serves as a quick reference to the calculations done in the SnoScience package. An example neural network (NN) is set up in the first section, and all subsequent sections use this example to explain the calculations. Additionally, the unit and integration tests for this package use this setup as well.
Setup
The network is set up as follows:
- 2 inputs (\(I\)) per sample, 2 samples in total.
- 2 neurons (\(N\)) in first hidden layer (\(H1\)).
- 3 neurons in second hidden layer (\(H2\)).
- 2 outputs (\(O\)) per sample, thus 2 neurons in output layer, and thus 2 losses (\(L\)).
- all neurons are initialised with weights and biases equal to \(\frac{1}{2}\) to simplify the calculations.
See the table below for naming conventions:
| Input | Hidden 1 | Hidden 2 | Output | Loss |
|---|---|---|---|---|
| \(I1\) | \(N11\) | \(N21\) | \(O1\) | \(L1\) |
| \(I2\) | \(N12\) | \(N22\) | \(O2\) | \(L2\) |
| \(N23\) |
The naming conventions for the neuron weights (\(W_N\)) and biases (\(B_N\)) are given below, with \(N22\) and \(O1\) as examples:
| \(N22\) | \(O1\) |
|---|---|
| \(W_{N22_1}\) | \(W_{O1_1}\) |
| \(W_{N22_2}\) | \(W_{O1_2}\) |
| \(B_{N22}\) | \(W_{O1_3}\) |
| \(B_{O1}\) |
The inputs and outputs of the samples are defined in the following matrices:
Initialisation of the neurons
Each neuron is initialised with a randomised weight vector and bias scalar with values between 0.00 and 0.01. The number of weights is determined by the number of neurons in the previous layer. The learning rate used in SGD is correlated to these initialised values, and using the 0.01 factor leads to commonly used learning rates between 0.00 and 0.05. Note that in the examples used on this page and during testing, this randomisation is not done!
Calculate inputs
The input for the neuron (\(X_N\)) is calculated with the following equation:
E.g. for \(N11\):
Calculate outputs
The output for the neuron (\(N\)) depends on the activation function used. For the sigmoid, it is calculated as follows:
E.g. for \(N11\):
Note that the naming convention for the neuron output is equal to the neuron itself. This is intentionally done to improve the readability of the total derivatives, which is further discussed in the derivatives section.
Calculate output derivates
Like the outputs themselves, the output derivatives for the neuron (\(N'\)) also depend on the activation function used. For the sigmoid, the derivative is calculated as follows:
E.g. for \(N11\):
Training the neurons
For SGD, the total derivative of the loss with respect to the neuron's weights (\(\frac{\delta{L}}{\delta{W_N}}\)) and its bias (\(\frac{\delta{L}}{\delta{B_N}}\)) need to be known. This is not a straightforward calculation, and is further discussed below. Once calculated, the training can be done using the following equations, with \(l\) corresponding to the learning rate:
Calculate total derivatives
As described above, \(\frac{\delta{L}}{\delta{W_N}}\) and \(\frac{\delta{L}}{\delta{B_N}}\) are required to implement SGD as an optimiser. These can be subdivided in the total derivative for the neuron \(\frac{\delta{L}}{\delta{N}}\), its weights \(\frac{\delta{L}}{\delta{W_N}}\), both of which are vectors, and its bias \(\frac{\delta{L}}{\delta{B_N}}\), which is a scalar. Both \(\frac{\delta{L}}{\delta{W_N}}\) and \(\frac{\delta{L}}{\delta{B_N}}\) depend on \(\frac{\delta{L}}{\delta{N}}\). Using \(N11\) as an example, the total derivative for the neuron can be formulated as follows using the chain rule:
The parts containing \(\frac{\delta{L1}}{\delta{O2}}\) and \(\frac{\delta{L2}}{\delta{O1}}\) have been omitted in the equation, as they always equate to 0. The total derivative for neuron's weights and bias can then be calculated with these equations:
Calculate partial derivatives
The partial derivatives within the total derivative have a pattern, namely:
Derivative \(\frac{\delta{L}}{\delta{O}}\)
This partial derivative (the loss derivative) always depends on the loss function used and the output of the network.
Derivative \(\frac{\delta{O}}{\delta{N}}\) or \(\frac{\delta{N_{a}}}{\delta{N_{b}}}\)
For these partial derivatives, the following holds:
To determine \(\frac{\delta{N22}}{\delta{N11}}\) for example, the first weight of \(N22\) has to be taken:
Derivative \(\frac{\delta{N}}{\delta{W_N}}\)
For this partial derivative, the following holds:
So, for \(N11\), which is in the first hidden layer:
For \(N22\), which is in the second hidden layer:
Derivative \(\frac{\delta{N}}{\delta{B}}\)
For this partial derivative, the following holds:
Reference calculations
Using the above equations and NN setup, the change in NN outputs after a training iteration (epoch) can be calculated. The calculations given below are used in the unit and integration tests as well.