# Basics of Deep Learning: Backpropagation by Byoungsung Lim

We make predictions on X on Line 155 and then compute the sum squared error on Line 156. The loss is then returned to the calling function on Line 159. This process is repeated until we reach the first layer in the network.

• The algorithm is used to effectively train a neural network through a method called chain rule.
• You can update them in any order you want, as long as you don’t make the mistake of updating any weight twice in the same iteration.
• The human brain is estimated to have about 10 billion neurons, each connected to an average of 10,000 other neurons.
• In the backward pass, the flow is reversed so that we start by propagating the error to the output layer until reaching the input layer passing through the hidden layer(s).

The delta for the current layer is equal to the delta of the previous layer, D[-1] dotted with the weight matrix of the current layer (Line 109). To finish off the computation of the delta, we multiply it by passing the activation for the layer through our derivative of the sigmoid (Line 110). We then update the deltas D list with the delta we just computed (Line 111). Between the input and output layers, there might be 0 or more hidden layers.

Backpropagation in neural network is a short form for “backward propagation of errors.” It is a standard method of training artificial neural networks. This method helps calculate the gradient of a loss function with respect to all the weights in the network. Backpropagation is a supervised machine learning algorithm that teaches artificial neural networks how to work. It is used to find the error gradients with respect to the weights and biases in the network. Gradient descent then uses these gradients to change the weights and biases. Backpropagation is a widely used algorithm for training artificial neural networks.

## Access the code to this tutorial and all other 500+ tutorials on PyImageSearch

The project builds a generic backpropagation neural network that can work with any architecture. To update the weight, we calculate the error correspond to each weight with the help of a total error. The error on weight w is calculated by differentiating total error with respect to w. Now, we will backpropagate this error to update the weights using a backward pass. And applies its activation function to calculate the output signals.

• Backpropagation in a neural network is designed to be a seamless process, but there are still some best practices you can follow to make sure a backpropagation algorithm is operating at peak performance.
• In deep learning, a set of linear operations between layers would be just a big linear function after all if without activation functions.
• Let’s now update the weights according to the calculated derivatives.
• The derivatives of the error W.R.T to the weights are saved in the variables gradw1 and gradw2.
• They are straightforward to implement and applicable for many scenarios, making them the ideal method for improving the performance of neural networks.
• Lines simply check to see if we should display a training update to our terminal.

Line 34 computes the output predictions for every data point in testX. The predictions array has the shape (450, 10) as there are 450 data points in the testing set, each of which with ten possible class label probabilities. Inserting a column of 1’s into our feature vector is done programmatically, but to ensure we understand this point, let’s update our XOR design matrix to explicitly see this taking place (Table 1, right). As you can see, a column of 1’s have been added to our feature vectors. In practice you can insert this column anywhere you like, but we typically place it either as (1) the first entry in the feature vector or (2) the last entry in the feature vector. I hope this example manages to throw some light on the mathematics behind computing gradients.

Our classification report demonstrates that we are obtaining ≈98% classification accuracy on our testing set; however, we are having some trouble classifying digits 4 and 5 (95% and 94% accuracy, respectively). Later in this book, we’ll learn how to train Convolutional Neural Networks on the full MNIST dataset and improve our accuracy further. The first phase of the backward pass is to compute our error, or simply the difference between our predicted label and the ground-truth label (Line 91). Since the final entry in the activations list A contains the output of the network, we can access the output prediction via A[-1]. In Jaderberg, Max, et al. “Decoupled neural interfaces using synthetic gradients.” International Conference on Machine Learning.

## Derivatives of the Prediction Error W.R.T Parameters

Backpropagation algorithms are crucial for training neural networks. They are straightforward to implement and applicable for many scenarios, making them the ideal method for improving the performance of neural networks. Deciding on the learning rate for training a backpropagation model depends on the size of the data set, the type of problem and other factors. That said, a higher learning rate can lead to faster results, but not the optimal performance. A lower learning rate produces slower results, but can lead to a better outcome in the end.

## Artificial Intelligence Algorithms: All you need to know

To further enhance your skills, I strongly recommend watching Stanford’s NLP series where Richard Socher gives 4 great explanations of backpropagation. The final step in a forward pass is to evaluate the predicted output s against an expected output y. Now carefully observe the neural network illustration from above. Also, it uses the matplotlib library to create 2 plots, showing how the predicted output and the error evolves by epoch.

A backpropagation algorithm can then more easily analyze the data, leading to faster and more accurate results. It might not make sense that all the weights have the same value again. However, training the model on different samples over and over again will result in nodes having different weights based on their contributions to the total loss. I have included a plot of the squared loss as well (Figure 5). Notice how our loss starts off very high, but quickly drops during the training process.

## Objective Function

From every layer, we calculate the gradients regarding the activation layer first. Then, the inner product of that gradient to the input values (z’) will be the gradient with respect to our weights. Also, the inner product of the gradient to the weights (w) will be the next passing gradient to the left.

## Data Structures and Algorithms

A supervised learning method enables a neural network to learn from a dataset by adjusting its weights and biases. When we are training the network, we are simply updating the weights so that the output result becomes closer to the answer. In other words, with a well-learned network, we can correctly classify an image to whatever class it really is. We calculate the gradients and gradually update the weights to meet the objectives. An objective function (aka loss function) is how we are going to quantify the difference between the answer and the prediction we make. With a simple and differentiable objective function, we can easily find the global minimum.

Usually, each neuron in the hidden layer uses an activation function like sigmoid or rectified linear unit (ReLU). This helps to capture the non-linear relationship between the inputs and their outputs. The neurons in the output layer also use activation functions like sigmoid (for regression) or SoftMax (for classification). Backpropagation is “backpropagation of errors” and is very useful for training neural networks. Backpropagation does not require any parameters to be set, except the number of inputs.

I know it’s a lot of information to absorb in one sitting, but I suggest you take your time to really understand what is going on at each step before going further. All too often I see developers, students, and researchers wasting their time, studying the wrong things, and generally struggling to backpropagation tutorial get started with Computer Vision, Deep Learning, and OpenCV. I created this website to show you what I believe is the best possible way to get your start. Notice how we are importing our newly implemented NeuralNetwork class. Again, these weight values are randomly sampled and then normalized.

The model is not trained properly yet, as we only back-propagated through one sample from the training set. Doing everything all over again for all the samples will yield a model with better accuracy as we go, with the aim of getting closer to the minimum loss/cost at every step. We do the delta calculation step at every unit, backpropagating the loss into the neural net, and find out what loss every node/unit is responsible for. In order to get the loss of a node (e.g. Z0), we multiply the value of its corresponding f’(z) by the loss of the node it is connected to in the next layer (delta_1), by the weight of the link connecting both nodes. All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. My mission is to change education and how complex Artificial Intelligence topics are taught.

Since we need to know the effect that each input variables make on the output result, the partial derivatives of f given x, y, or z are the gradients we want to get. Then, by the chain rule, we can backpropagate the gradients and obtain each local gradient as in the figure above. The theory behind machine learning can be really difficult to grasp if it isn’t tackled the right way. One example of this would be backpropagation, whose effectiveness is visible in most real-world deep learning applications, but it’s never examined. We implemented our backpropagation algorithm using the Python programming language and devised a multi-layer, feedforward NeuralNetwork class.

Scroll to Top
Scroll to Top