SDS 568: PaLM: Google's Breakthrough Natural Language Model — with Jon Krohn

Parameter Expansion

Google's PaLM model represents a significant leap in natural language processing with its 540 billion parameters, vastly surpassing previous models like GPT-3. Jon Krohn highlights that PaLM achieved state-of-the-art results on 28 out of 29 English language tasks, showcasing its prowess in areas like question answering and natural language inference 1. The model's capacity to handle multilingual tasks and explain jokes it hasn't encountered before is particularly impressive.

The joke is that the whale is able to communicate between the two groups of whales, but the speaker is pretending that the whale is able to communicate between the two groups of TPUs.

--- Jon Krohn

Krohn speculates that future expansions could see PaLM's parameters reach trillions, potentially unlocking even more advanced capabilities 1.

   

Emerging Behaviors

The potential for PaLM to exhibit new behaviors as its parameters increase is a fascinating prospect. Jon Krohn notes that larger models tend to display emergent behaviors, and PaLM's current capabilities already hint at this potential 1. For instance, it can convert C code to Python and solve complex programming problems despite limited training data.

Given how large language models like PaLM reliably exhibit more and more emergent behaviors, the larger they become, I'd say it's a good bet that we'll be hearing about mind-blowing new PaLM feats soon.

--- Jon Krohn

As PaLM continues to grow, it may redefine what is possible in natural language processing and beyond 1.