657: How to Learn Data Engineering — with Andreas Kretz (@andreaskayy)

Roles & Responsibilities

Data engineering is a critical role that bridges the gap between raw data and actionable insights. Jon Krohn and Andreas Kretz discuss how data engineers manage data pipelines, ensuring data is clean and structured for data scientists and analysts 1. Senior data engineers often lead teams, breaking down complex problems and guiding junior engineers through tasks 2. Andreas emphasizes the importance of aligning one's interests with the role, noting that those with a computer science background often gravitate towards data engineering due to its focus on tools and software development 2.

It's not just you. You have a team. You basically help the whole team, work on your projects, work on your goals.

--- Andreas Kretz

Choosing between data engineering and other roles like machine learning engineering often depends on one's career aspirations and desire for specialization 2.

   

Key Skills

Key skills for data engineers include a strong foundation in computer science and proficiency with essential tools and platforms. Andreas Kretz highlights the importance of understanding relational databases, APIs, and data processing frameworks like Apache Spark 3. He advises against trying to learn every tool, instead recommending a focus on understanding common templates and processes 4.

You need to understand. Okay, what's the usual template? How is it this is usually going to work?

--- Andreas Kretz

Emerging tools like Snowflake and Databricks are becoming ubiquitous in the field, offering robust solutions for data management and processing 4.