The discussion highlights the differences between embeddings in NLP and structured data, emphasizing the simpler nature of the latter. Mark shares insights on cleaning messy geocoding data from Toronto's streetcar system and how categorical values are transformed into integer IDs for model training. The innovative approach of automatically defining model layers based on input columns is also explored, showcasing a streamlined process for building predictive models.