771: Gradient Boosting: XGBoost, LightGBM and CatBoost — with Kirill Eremenko

XGBoost

XGBoost, short for Extreme Gradient Boosting, has become a staple in competitive machine learning since its introduction in 2014 by Tianqi Chen. Kirill Eremenko highlights its rapid adoption, noting that in 2015, 17 out of 29 winning solutions on Kaggle utilized XGBoost, showcasing its effectiveness and popularity 1. The algorithm was developed to transition from theoretical to applied gradient boosting, making it more computationally efficient and resource-friendly 2. Jon Krohn and Kirill discuss its continued relevance, with XGBoost remaining one of the top non-deep learning algorithms used today 1.

   

LightGBM

LightGBM, introduced by Microsoft in 2017, is renowned for its speed, often considered 20 times faster than XGBoost, albeit with some trade-offs in accuracy 3. Kirill explains that LightGBM's efficiency stems from techniques like exclusive feature bundling, which reduces the number of columns by focusing on sparse data 4. This approach allows LightGBM to handle large datasets swiftly, making it a preferred choice for many Kaggle competitions 3.

   

CatBoost

CatBoost, developed by Yandex in 2017, excels in handling categorical data, a common challenge in machine learning 5. Kirill emphasizes its use of ordered target encoding and symmetric trees, which enhance its performance with categorical features 6. Unlike other algorithms, CatBoost automatically processes categorical variables, eliminating the need for one-hot encoding, and making it a powerful tool for datasets with categorical features 5.