The Limits of NLP

Model Scaling

The discussion on model scaling reveals the challenges and considerations involved in expanding NLP models to 11 billion parameters. Colin Raffel explains that while scaling up models can improve performance, it also introduces significant costs and hardware limitations. He notes that the increase from 100 million to 11 billion parameters is a two orders of magnitude leap, which is impressive but not without its drawbacks 1. Raffel highlights that the limits explored in their paper pertain to how large models can be made and the amount of data they can be trained on with current hardware 2.

Scaling is not the most satisfying solution to these problems. It's great that we can get good performance by making the model bigger, but currently it's still expensive to run inference on our biggest model.

--- Colin Raffel

He emphasizes that while larger models achieve better results, they are not yet practical for all applications, such as running on mobile devices.

   

Task Difficulty

Creating and solving challenging NLP tasks is crucial for pushing model capabilities beyond current limits. Colin Raffel mentions that many tasks, such as those in the GLUE and SuperGLUE benchmarks, have been saturated, with models reaching or surpassing human performance 3. However, the NLP community continues to design more difficult tasks to test these models further. Raffel describes an innovative approach called the adversarial natural language inference benchmark, where humans interact with models to identify weaknesses and improve them iteratively.

We develop new ideas, new approaches, scale up existing approaches to attack problems. We might be able to get good performance on those problems, and the community comes back and gives us harder problems to work on.

--- Colin Raffel

This cyclical process of creating and solving tasks helps identify gaps in model abilities and drives continuous improvement.