SDS 568: PaLM: Google's Breakthrough Natural Language Model — with Jon Krohn

Model Capabilities

Jon Krohn introduces Google's PaLM, a groundbreaking natural language model that excels in a wide array of NLP tasks. PaLM achieved state-of-the-art results on 28 of 29 English language tasks, surpassing models like GPT-3 in areas such as question answering, sentence completion, and natural language inference 1. It even performed well on multilingual tasks, despite only 22% of its training data being non-English. A remarkable aspect of PaLM is its ability to explain jokes it hasn't encountered before, showcasing its advanced understanding of context and language nuances 1.

PaLM can even explain brand new jokes that it couldn't possibly have learned from its Internet-based training data.

--- Jon Krohn

Beyond language, PaLM can solve programming questions, converting C code to Python, despite having significantly less Python training data compared to other models 1.

   

Development and Scale

The development of PaLM marks a significant leap in AI model scaling, with its 540 billion parameters making it three times larger than GPT-3. Jon Krohn highlights that PaLM leverages Google's Pathways approach, allowing for shared concept-specific modules across various computational pathways 1. This innovation enables PaLM to operate at an unprecedented parameter scale, setting a new benchmark in AI model development.

The key innovation within PaLM is scaling up this powerful pathways modeling approach to half a trillion parameters.

--- Jon Krohn

Future iterations of PaLM could potentially exceed a trillion parameters, promising even more advanced capabilities and emergent behaviors 1.