Machine Learning Python

Linear Regression using Stochastic Gradient Descent in Python

Pinterest LinkedIn Tumblr
Stochastic gradient descent plot with python

In today’s tutorial, we will learn about the basic concept of another iterative optimization algorithm called the stochastic gradient descent and how to implement the process from scratch. You will also see some benefits and drawbacks behind the algorithm.

If you aren’t aware of the gradient descent algorithm, please see the most recent tutorial before you continue reading.

The Concept behind Stochastic Descent Descent

One of the significant downsides behind the gradient descent algorithm is when you are working with a lot of samples. You might be wondering why this poses a problem.

The reason behind this is because we need to compute first the sum of squared residuals on as many samples, multiplied by the number of features. Then compute the derivative of this function for each of the features within our dataset. This will undoubtedly require a lot of computation power if your training set is significantly large and when you want better results.

Considering that gradient descent is a repetitive algorithm, this needs to be processed for as many iterations as specified. That’s dreadful, isn’t it? Instead of relying on gradient descent for each example, an easy solution is to have a different approach. This approach would be computing the loss/cost over a small random sample of the training data, calculating the derivative of that sample, and assuming that the derivative is the best direction to make gradient descent.

Along with this, as opposed to selecting a random data point, we pick a random ordering over the information and afterward walk conclusively through them. This advantageous variant of gradient descent is called stochastic gradient descent.

In some cases, it may not go in the optimal direction, which could affect the loss/cost negatively. Nevertheless, this can be taken care of by running the algorithm repetitively and by taking little actions as we iterate.

Applying Stochastic Gradient Descent with Python

Now that we understand the essentials concept behind stochastic gradient descent let’s implement this in Python on a randomized data sample.

Open a brand-new file, name it linear_regression_sgd.py, and insert the following code:

Let’s get started by importing our required Python libraries from Matplotlib, NumPyand Seaborn.

To correctly apply stochastic gradient descent, we need a function that returns mini-batches of the training examples provided. This next_batch function takes in as an argument, three required parameters:

  • Features: The feature matrix of our training dataset.
  • Labels: The class labels link with the training data points.
  • batch_size: The portion of the mini-batch we wish to return.

Along with this, the batch size is set to one by default. Nonetheless, we commonly make use of mini-batches that are greater than one.

Note: A few other values include 32, 64, 128, and 256.

So, why are these numbers the typical mini-batch size standard? Making use of a batch size greater than one does help reduce the difference (variance) in the parameter update and helps bring about a more stable convergence.

Within the stochastic_gradient_descent function, we performed some initialization.

For an extra thorough evaluation of this area, please see last week’s tutorial.

Now below is our actual Stochastic Gradient Descent (SGD) implementation in Python:

Let’s start by looping through our desired number of epochs.

Next, to keep track of the cost throughout each batch processing, let’s initialize a batch_epoch_cost_list, which we will certainly use to calculate the average loss/cost over all mini-batch updates for each epoch.

Within the inner loop exists our stochastic gradient descent algorithm. The significant difference between what we discovered last week, and what you are reading currently, is that we aren’t going over all our training samples at once. We are looping over these samples in mini-batches as explained above.

For each mini-batch returned to us by the next_batch function, we take the dot product between our feature matrix and weights matrix. Then compute the error between the estimated value and actual value.

When we evaluate the gradient for the current batch, we have the gradient which can then update the weight matrix \(W\) by simply multiplying the gradient scaled by the learning rate subtracted from the previous weight matrix.

Again, for a more thorough, detailed explanation of the gradient descent algorithm, please see last week’s tutorial.

If you are having difficulties understanding the concept behind gradient descent, please refer to this tutorial.

For the final step, to walk you through what goes on within the main function, where we generated a regression problem, on lines 90 – 92. We have a total of 1000 data points, each of which is 5D.

Our main objective is to correctly map these randomized feature training samples \((1000×5)\) (a.k.a dependent variable) to our independent variable \((1000×1)\).

On lines 95-98, we pass the feature matrix, target vector, learning rate, number of epochs, and the batch size to the stochastic_gradient_descent function, which returns the best parameters and the saved history of the lost/cost after each iteration.

The last block of code from lines 101 – 105 aids in envisioning how the cost adjusts on each iteration.

To check the cost modifications from your command line, you can execute the following command:

Stochastic gradient descent plot with python
Figure 1: Visualization of the cost function changing overtime

Advantages of Stochastic Gradient Descent

Usually in practice, stochastic gradient descent is often preferred if we have:

  1. Lots of data: Stochastic gradient descent has better speed properties because we don’t have to touch all \(m\) datapoints before updating our \(\theta\) parameter once.
  2.  Computationally fast:  Updating our parameter is done quickly after seeing a few data points before modifying it.

Drawbacks of Stochastic Gradient Descent

Nonetheless, since the procedure is random, it can be:

  1. Harder to Debug: Difficulty in assessing convergence.
  2. Harder to know when to stop: Stopping conditions may be harder to evaluate.

Feel free to click the “Click to Tweet” button if you enjoyed the post.

Check out the post on Stochastic Gradient Descent (SGD) with Python. Click To Tweet

Conclusion

To conclude this tutorial, you learned about Stochastic Gradient Descent (SGD), a common extension to the gradient descent algorithm in today’s blog post. You’ll see SGD used in nearly all situations instead of the original gradient descent version. You learned:

  • The Concept behind Stochastic Descent Descent Algorithm
  • How to Apply the Algorithm yourself in Python
  • Benefits of using the Algorithm
  • Then a few drawbacks behind the Algorithm

Do you have any questions about this post or Stochastic Gradient Descent? Leave a comment and ask your question. I’ll do my best to answer.

To get access to the source codes used in all of the tutorials, leave your email address in any of the page’s subscription forms.

Reference

Further Reading

We have listed some useful resources below if you thirst for more reading.

Course

Books

To be notified when this next blog post goes live, be sure to enter your email address in the form!

6 Comments

  1. Very nice post. I just stumbled upon your weblog and wished to say that I have really enjoyed surfing around your
    blog posts. In any case I will be subscribing to your mailing list feed and I hope you write again soon!

  2. I just like the helpful information you supply in your articles.
    I’ll bookmark your blog and take a look at again right here regularly.
    I am slightly certain I’ll be told many new stuff proper right here!
    Best of luck for the next!

Write A Comment