Roland discusses the advantages of transformers in capturing longer contexts compared to traditional recurrent architectures. He highlights the intriguing concept of pseudo-recurrence in models that appear to rely on past actions while often resorting to pattern matching. This raises questions about their true capabilities, particularly regarding length generalization, as recent studies reveal potential limitations in these models.
