657: How to Learn Data Engineering — with Andreas Kretz (@andreaskayy)

Relational Databases

Relational databases remain a cornerstone in data engineering, despite their long history. Andreas Kretz emphasizes their continued relevance, noting that data often originates from these systems and is accessed via SQL, a critical skill for data engineers 1. Understanding SQL and coding in Python are foundational skills for those entering the field, as they enable engineers to manage data pipelines effectively 2. Andreas highlights the importance of knowing how data is used in machine learning processes, which enhances collaboration with data scientists 2.

A lot of stuff that a data engineer needs to know is relational databases. I know it sounds very boring, man. These relational databases have been here for 30 years.

--- Andreas Kretz

This understanding allows engineers to optimize data structures for analysis and application, ensuring efficient data handling and processing.

   

Cloud Platforms

Cloud platforms like AWS, Azure, and GCP are pivotal in modern data engineering. Andreas Kretz suggests AWS as a starting point due to its widespread use, but emphasizes the importance of choosing a platform based on industry needs 3. He advises that understanding one platform can facilitate learning others, as the core concepts often overlap 3. Tools like DBT, which act as abstraction layers over data warehouses, are gaining popularity for their ability to simplify complex processes 4.

AWS is the one that is most used out there. So it's a fairly safe bet to go with AWS.

--- Andreas Kretz

Andreas notes that certifications are valuable for education and proving specific knowledge, though they are not always required for employment.