Dan discusses the impact of constraints on transformer models, noting that while they limit the types of transformers that can be learned, the framework remains quite expressive. He highlights the challenge of optimization, indicating that even with these limitations, effective solutions can be elusive. The conversation emphasizes the importance of exploring this constrained space rather than starting from a traditional transformer model.
