Learning rate schedules. A picture

Time for a short lesson just talking about the learning rate without seeing it doesn’t inform us as

well as seeing a graph as they say a picture is worth a thousand words.

We can see a graph of the loss as a function of the number of epochs.

So far we have only seen this case.

We used a small learning rate which reached the goal but slowly as we said a high learning rate would

minimize the loss fast.

But only to a certain extent.

Then it starts oscillating and the loss stops moving a high learning rate would not even minimize the

loss.

The cost would rather explode upwards as seen in the graph.

Finally a well-selected learning rate such as the one defined by the exponential schedule would minimize

the loss much faster than a low learning rate.

Moreover it would do so more accurately than a high learning rate.

Naturally we are aiming at the good learning rate.

Problem is we don’t know what this learning rate is for our particular data model.

One way to establish a good learning rate is to plot the graph we’re showing you now for a few learning

rate values and pick the one that looks the most like the good curve.

Note that high learning rates may not minimize the loss a low learning rate eventually converges with

a good learning rate.

But the process will take much longer.

All right I’m sure this was useful and clear doubts you had.

Take care and thanks for watching.