SDS 595: Data Engineering 101 — with Joe Reis and Matt Housley

Communication

Effective communication with downstream stakeholders is crucial in data engineering. Matt Housley explains that traditionally, application developers create data schemes without much consideration for downstream needs, leading to inefficiencies. He advocates for bidirectional communication between application developers and data engineers to ensure better outcomes, such as real-time dashboards for users 1. Joe Reis adds that poor communication often results in data engineers merely going through the motions without understanding the end goals 1.

If your cultural norm is to throw things over the wall, that's what you're going to do. This happens a lot of places. This is sort of, I would say the default, because it's just like, not my problem, not my job.

--- Joe Reis

Joe emphasizes that communication is one of the most underutilized tools in a data engineer's toolbox, yet it's essential for solving 90% of the problems data teams face 2.

   

Collaboration

Cross-functional collaboration between data engineers, data scientists, and other teams is vital for successful data projects. Joe Reis and Matt Housley discuss how data scientists often need to take on data engineering tasks when the necessary infrastructure isn't in place 3. They highlight that understanding one's role and the company's data maturity is crucial for effective collaboration 4.

If you're a data scientist who's been hired by a company that's pretty low in the data maturity, and you're the only data person there, either it's going to be you or the software engineer that ends up building the systems that will support data science.

--- Joe Reis

Joe and Matt also note that data scientists with a background in applied math or statistical approaches may find data engineering tasks more aligned with their skills, facilitating better collaboration and project outcomes 4.