Pablo shares his journey of training language models for generating song lyrics, highlighting the transition from recurrent neural networks to transformer models. Despite achieving good perplexity scores, the model often produced repetitive and uninspired outputs, revealing the complexities of teaching a machine to create engaging and poetic lyrics. The discussion emphasizes the balance between structure and creativity in language generation.
