Nicholas Carlini (Google DeepMind)

Chess Insights

Nicholas Carlini from Google DeepMind shares insights into the unexpected capabilities of large language models (LLMs) in playing chess. He explains that LLMs, trained to predict the next token, can make valid and high-quality chess moves without explicit instructions about the game's rules or board setup. This ability suggests that the models have an internal understanding of the game, which Carlini finds fascinating.

The fact that it can make valid moves almost always means that it must in some sense have something internally that is accurately modeling the world.

--- Nicholas Carlini

Despite these capabilities, Carlini notes that attempts to adjust models' performance by altering parameters, such as Elo ratings, have not yielded significant improvements, indicating the complexity of training models to match specific skill levels 1 2.

   

Post-Training Effects

Carlini discusses the dynamics of language models post-training, particularly focusing on Reinforcement Learning from Human Feedback (RLHF). He highlights that post-training helps align models with user expectations, making them more responsive to varied queries. However, this process can disrupt the model's initial calibration, affecting its probability accuracy.

When you do the post-training process, the calibration gets all messed up and it doesn't have this behavior anymore.

--- Nicholas Carlini

Carlini also draws parallels between programming languages like Rust and LLMs, emphasizing the need to adapt to their unique paradigms to fully leverage their capabilities 3 4.