#128 - Michael Babineau and Kevin Hale

Data Cleaning

Second Measure employs sophisticated data cleaning techniques to handle the complexities of transaction data. Michael Babineau explains that their approach involves building a pipeline that ingests raw data and outputs useful information, using machine-based methods to tackle issues like entity resolution and location inaccuracies 1. This process is crucial because credit card statements often contain ambiguous merchant identifiers, leading to a cardinality problem where multiple variants exist for a single merchant 2.

We've basically had to build two different products. One is this pipeline, which ingests raw transactional data and then output something useful.

--- Michael Babineau

The team also focuses on debiasing their data to ensure it represents a broader population, despite the inherent limitations of their consumer panel 1.

   

Classification Challenges

Classification issues arise from the variability in merchant identifiers and consumer behaviors, complicating the accurate categorization of transaction data. Michael Babineau highlights that human errors and inconsistencies in point-of-sale systems contribute to this problem, as different franchises may use varied identifiers for the same brand 3. This lack of standardization leads to frustration among users who must repeatedly correct misclassified data.

The problem is, depending on how the transaction is processed, the apostrophe that you expected to appear in McDonald's could be a space, it could be a star, it could be deleted.

--- Michael Babineau

Additionally, Kevin Hale notes that the limited incentive for individuals to classify their own data further exacerbates these challenges 3.