Gradient Descent

Gradient descent is most common taught during Calculus in school.

Gradient descent is an algorithm that numerically estimates where a function outputs its lowest values. That means it finds local minima, but not by setting:

\begin{equation} \nabla f = 0 \end{equation}

Instead of finding minima by manipulating symbols, gradient descent approximates the solution with numbers. Furthermore, all it needs in order to run is a function's numerical output, no formula required.

This distinction is worth emphasizing because it's what makes gradient descent useful. If we had a simple formula like: \[f(x) = x^2 - 4x\]

then we could easily solve: \[\nabla f = 0\]

to find that: \[x = 2\]

minimizes: \[f(x)\]

Or we could use gradient descent to get a numerical approximation, something like: \[x \approx 1.99999967\]

Both strategies arrive at the same answer.

This is just a simple example however and gradient descent becomes a lot more useful when we are working with complex formulas. To learn more about gradient descent visit: Khan Academy.