Adji discusses the implications of using dropout as a regularization method for LSTMs, suggesting it may define a new model class for sequential data. He highlights that while traditional regularization often reduces parameters, injecting noise allows for a high-capacity network that avoids memorization of data. The conversation also touches on the computational complexity of noise-based approaches compared to tying input and output weights.
