The Rise of Modern Data Management // Chad Sanderson // #226
Topics covered
Popular Clips
Episode Highlights
Data Complexity
Managing data complexity within companies presents significant challenges, particularly in data extraction and transformation. explains that data producers often focus on their immediate needs, unaware of how their data might be used by others in the organization 1. This lack of awareness can lead to issues when data is extracted and transformed by other teams, potentially breaking downstream processes. Sanderson highlights the importance of understanding data dependencies, which resemble a supply chain, where any break can have a ripple effect throughout the organization 2.
If you go to any application developer and you ask them, do you know how your data is being used and you know where dependencies on you exist and what it's being used for and how valuable it is to the company, most of these developers are going to say, no, I have no clue.
---
He emphasizes the need for structured data management to ensure data is accessible and fit for purpose 3.
Change Management
Change management in data systems is fraught with challenges, as points out, due to the lack of established systems akin to DevOps for software 4. Unlike software changes, data changes often occur without human review or feedback, leading to potential issues that are only detected after the fact. Sanderson notes the absence of integration testing for data, which complicates the detection of data flow issues 5.
Data changes just as frequently as software changes. If I'm an engineer and I'm making a change to my database, that's data changing.
---
He stresses the importance of implementing systems for change detection and management to prevent disruptions in data workflows 6.
Data Lineage
Advanced data lineage plays a crucial role in tracking and managing changes across datasets. describes this process as creating a graph that shows how data flows through various tables and views within a company 7. This lineage is essential for understanding data dependencies and ensuring data quality. Sanderson envisions a future where data management is as integral to workflows as security is today, emphasizing the need for clear data lineage to prevent quality issues 8.
I think dev data ops will become a term that most DevOps teams and most engineering teams are using.
---
He predicts a shift towards treating data as a product, requiring quality measures from inception to ensure its value and reliability 8.