The Complexity of Learning Neural Networks

Achievements

Deep learning has achieved remarkable feats, notably in image classification, where algorithms now surpass human capabilities. John Wilmes highlights how machines can recognize different dog breeds more accurately than humans, despite lacking an understanding of what a dog is 1. This success extends to other areas, such as the game of Go, where AI has outperformed human champions 1. However, Wilmes emphasizes that while neural networks can approximate any continuous function, the practical utility of such representations remains a challenge 2.

The universal approximation theorem says that any continuous function can be represented to an arbitrarily high degree of accuracy using a neural network.

--- John Wilmes

The quest for new architectures and neural units continues, as researchers seek breakthroughs that could redefine the limits of neural network capabilities 2.

   

Vulnerabilities

Despite their impressive capabilities, deep learning models are not without flaws. John Wilmes discusses the phenomenon of fooling images, where slight, imperceptible noise can trick classifiers into misidentifying images 3. This vulnerability raises concerns about the robustness and reliability of these models, suggesting that they may not be learning as effectively as assumed.

You can add a little bit of noise to the image in a clever way and get the classifier to say whatever you want.

--- John Wilmes

Wilmes argues for the importance of formal guarantees in model correctness, highlighting the need for ongoing research to address these limitations and improve model defenses 3.