815: DataFrame Operations 100x Faster than Pandas — with Marco Gorelli

Techniques

Forecasting competitions often reward simplicity and robust validation techniques. Jon Krohn and Marco Gorelli emphasize the value of using simpler models to avoid overfitting and the critical role of cross-validation in estimating model performance on unseen data 1. Marco highlights that in time series forecasting, it's essential to ensure that training data does not include future information 1.

The most important thing isn't using the most unusual model, but having a good way of cross-validating your data.

--- Marco Gorelli

Marco also shares that the biggest benefit of participating in Kaggle competitions is learning to perform cross-validation effectively 2.

   

Experiences

Marco's journey in data science and forecasting competitions is filled with valuable lessons and unexpected successes. He recounts his transition from mathematics to data science and eventually to software engineering, driven by his passion for open-source projects 3. Participating in competitions like the M5, where he had to forecast Walmart sales, provided him with practical skills and insights 3.

I can tell you about how to beat other competitors in financial forecasting competitions, but not necessarily how to do trading.

--- Marco Gorelli

Despite not having a finance background, Marco's simple approach in the M6 competition earned him a top spot and a significant prize 4.