Trends in Deep Learning with Jeremy Howard - TWiML Talk #214

Fast Training

Jeremy Howard highlights significant advancements in accelerating neural network training, making it more accessible and cost-effective. He discusses the impact of the Stanford Dawnbench competition, which spurred interest in training large models quickly and cheaply, achieving remarkable results like training ImageNet in just 18 minutes using AWS 1. Techniques such as Leslie Smith's super convergence and one cycle scheduling have been pivotal in these advancements. Progressive resizing, which involves training on smaller images before scaling up, has also contributed to faster training times 2.

One of the main works towards increasing accessibility of deep learning is making it faster and easier to train neural nets across different areas.

--- Jeremy Howard

These innovations are making deep learning more accessible to developers and data scientists worldwide.

   

Normalization

Normalization techniques like batch norm and spectral norm are crucial for smoother loss landscapes and improved training stability. Jeremy Howard explains that while batch norm was initially thought to address covariate shift, it actually smooths the loss surface, enhancing training efficiency 3. Spectral norm and weight norm, now integrated into PyTorch, further stabilize training by focusing on scaling rather than covariate shift. Visualizing neural landscapes has revealed that much of the variation in gradient updates occurs in just two dimensions, simplifying the training process 4.

Batch norm doesn't help with covariate shift; it makes the loss surface smoother, which makes perfect sense.

--- Jeremy Howard

These insights are leading to more resilient models with fewer hyperparameters.