Amelia & Filip — How Pandora Deploys ML Models into Production

Debugging

Debugging machine learning models at Pandora involves a meticulous process to ensure reliability and performance. Filip Korzeniowski emphasizes starting with simple models and gradually increasing complexity to avoid errors. He notes, "It's difficult to debug models because there is no right way. And if you make a bug in your neural network training code, it will still mostly work. It's not like it will crash and burn. It will work, but worse."

It's difficult to debug models because there is no right way. And if you make a bug in your neural network training code, it will still mostly work. It's not like it will crash and burn. It will work, but worse.

--- Filip Korzeniowski

Filip shares a challenging experience with embedding networks, where data masking led to incomplete negative sampling, complicating the training process 1 2.

   

Optimization

Tuning and optimizing models at Pandora is a blend of intuition and structured approaches. Filip Korzeniowski describes using a hybrid method for hyperparameter search, starting with known configurations and iterating on data features. He explains, "In academia, you have a set dataset, but in industry, like at Pandora, you have to think about which data makes sense to use."

In academia, you have a set dataset, but in industry, like at Pandora, you have to think about which data makes sense to use.

--- Filip Korzeniowski

Amelia Nybakke adds that Pandora's ensemble recommender system uses multiple models to propose song candidates, with a microservice evaluating them in real-time 3 4.

   

Validation

Validation and testing are crucial for maintaining model accuracy and enhancing user experience at Pandora. Amelia Nybakke explains that daily model deployments ensure the latest data is used, reducing data drift concerns. She states, "We always run an A/B test. We never swap anything in without making sure that it's moving metrics in the right way."

We always run an A/B test. We never swap anything in without making sure that it's moving metrics in the right way.

--- Amelia Nybakke

Filip highlights the importance of communicating progress to non-technical teams, noting that their infrastructure allows tracing user interactions back to individual models, translating improvements into tangible business metrics 5 6.