Momentum and Clipping

Han discusses the importance of accumulating velocity over gradients to enhance accuracy in image classification, reducing loss from 1.5% to 0.36%. However, speech recognition faced challenges with convergence, prompting a shift in the sequence of gradient clipping and summation. This adjustment ultimately improved accuracy loss from 3% to 2% in speech recognition tasks.