Shubho discusses the critical design choices in neural networks regarding low precision for weights and gradients, emphasizing the sensitivity of recurrent networks to floating point errors. He highlights the challenges posed when transitioning algorithms from CPU to GPU, where slight variations in floating point calculations can significantly impact model performance. The conversation also touches on the evolving capabilities of GPUs to handle different precision types, catering to various fields from gaming to deep learning.
