Episode 395: Katharine Jarmul on Security and Privacy in Machine Learning

Differential Privacy

Differential privacy is a crucial technique in safeguarding personal data within machine learning models. Katharine Jarmul explains that differential privacy involves adding noise to data during the training process, which helps in quantifying privacy loss and ensuring that individual data points are not easily identifiable 1. However, excessive noise can reduce model accuracy, posing a challenge in balancing privacy and performance. She highlights the importance of understanding the specific privacy threats when releasing datasets and suggests methods like k-anonymization alongside differential privacy to mitigate risks 2.

Differential privacy allows us to try quantify privacy loss. So we're able to say, hey, you and your data, we're going to quantify your loss over time.

--- Katharine Jarmul

Collaboration with frameworks like TensorFlow Federated is essential for developing privacy-preserving machine learning solutions.

   

Privacy Attacks

Privacy attacks in machine learning are becoming increasingly sophisticated, with techniques like model inference and inversion attacks posing significant threats. Katharine Jarmul describes how these attacks can extract sensitive information from models, such as determining if an individual's data was part of the training set or reconstructing images from facial recognition models 3. Real-world examples, like the Netflix data correlation attack, demonstrate how seemingly anonymized data can be linked to individuals by correlating it with other datasets 4.

Humans tend to think of privacy in form of trust. When the average person is asked, even if they have public settings on Facebook, you know, who's reading your Facebook? They think my friends and family.

--- Katharine Jarmul

These examples underscore the need for robust privacy measures and the importance of designing systems with privacy as a default setting.