Deep Learning: Modular in Theory, Inflexible in Practice with Diogo Almeida - #8

Feature Engineering

Diogo Almeida, a senior data scientist, shares his approach to feature engineering in Kaggle competitions. He explains the complexity of generating 50,000 features by experimenting with various combinations of metrics and machine learning algorithms. This process involved converting categorical data into numerical forms using techniques like PCA and clustering, which allowed him to tackle complex problems effectively 1.

You can imagine exponential growth when you're just trying every combination of this. With every combination of this.

--- Diogo Almeida

His methodology also included using a feature selection algorithm to refine the features, which proved beneficial in generalizing solutions to new problems 2.

   

Winning Strategies

In discussing his winning strategies, Diogo Almeida highlights the importance of simplicity and confidence in his approach. He relied heavily on a boosted decision tree model, which was so effective that he maintained a significant lead throughout the competition without needing to ensemble other models 3. This approach was not only efficient but also robust against overfitting, as evidenced by his final victory despite competitors' attempts to catch up.

I had a model that took a week to train, like I said, and I only had one week left for the competition.

--- Diogo Almeida

Almeida's strategy also involved leveraging machine learning to automatically engineer informative variables, allowing him to focus on building a powerful model rather than hand-engineering features 4.