Kyunghyun Cho: Neural Machine Translation, Language, and Doing Good Science

Fine-Tuning

Fine-tuning models presents a complex set of challenges, particularly in maintaining optimization stability. Kyunghyun Cho highlights the mystery of how pretraining sets the stage for effective fine-tuning, despite the different datasets and loss functions involved 1. He emphasizes the importance of staying near the initial point provided by pretraining while allowing for necessary exploration to solve problems effectively 2. This balance is crucial, as Cho notes, "It's a weird trade-off between wanting to stay near the initial point, but at the same time, we need to go substantially further away in order to solve the problem well."

   

Adapter Fusion

Adapter Fusion emerges as a promising solution to the challenges of multitask learning and transfer learning. Cho explains that this method involves training individual adapters for each task and then fusing them to avoid destructive interference 3. This approach allows for effective multitask learning without the need for extensive hyperparameter tuning. Cho appreciates the elegance of this solution, stating, "I like this kind of algorithm, where if there is an issue, let's design an algorithm that's going to bypass or address it perfectly."

   

Pretraining

Pretraining and initialization play pivotal roles in the success of neural networks. Cho recounts his early struggles with training Boltzmann machines, highlighting the importance of correct initialization 4. He notes that pretraining provides a starting point for optimization, which is crucial for effective model training 5. Cho reflects on the evolution of these techniques, saying, "We need to keep on ensuring that some people do look into some of these small questions that are just hanging there without being answered."