[MINI] Gradient Descent

Local Optima

Gradient descent, a popular optimization algorithm, can sometimes lead to getting stuck in local optima, akin to being trapped in a valley. Kyle Polich explains that in complex problems, especially in high-dimensional spaces, it's challenging to determine if you're in a local optima. He describes it as being at the bottom of a hole where every direction seems to go up, yet it's not the global best solution 1.

The valley could always go out to the ocean. I guess I'm thinking of basically like a pit.

--- Kyle Polich

To address this, algorithms adjust the step size or learning rate, taking smaller steps as they approach a solution to avoid overshooting the optimal point.

   

Overshooting

The tendency of gradient descent to overshoot solutions is likened to hiking scenarios where one might zigzag down a valley. Kyle Polich illustrates how, in complex spaces, careful navigation is crucial to avoid missing optimal paths 2. He explains that gradient descent involves making quick jumps based on local gradients, which can lead to overshooting valleys that are preferable 3.

If you just got stuck in your mind like, okay, I'm going to walk 50 yards ahead after I've made a course adjustment.

--- Kyle Polich

This highlights the importance of reassessing direction frequently to find the best path forward, akin to stochastic gradient descent.