Episode 395: Katharine Jarmul on Security and Privacy in Machine Learning

API Security

API security is crucial in safeguarding machine learning models from data and model theft. Katharine Jarmul emphasizes the importance of understanding the threat model to protect against potential breaches. She suggests that a proactive approach to API security, such as implementing rate limiting and authentication, can serve as a foundational step in securing these systems 1.

Shutting down your APIs or obfuscating them in different ways and protecting them in different ways is a useful initial step.

--- Katharine Jarmul

Additionally, Justin Beyer highlights the need for validating incoming data to ensure it meets expected distributions and values, which can prevent adversarial attacks 2.

   

Adversarial Threats

Adversarial threats pose significant challenges to machine learning models, particularly through poisoning and adversarial examples. Katharine Jarmul explains that adversarial attacks can trick models into misclassifying inputs, such as making a turtle appear as a rifle 3. These attacks exploit the complexity of neural networks, which often have more weights than necessary, allowing for manipulation through carefully crafted noise.

We can truly, let's say, quote unquote, trick the model into classifying, for example, that turtle as a rifle.

--- Katharine Jarmul

She also notes that adversarial learning attacks have grown into a major research area, with conferences dedicating entire tracks to understanding and mitigating these vulnerabilities 4.

   

Model Access

Model access security is a critical concern, particularly with the rise of black box attacks. Katharine Jarmul describes black box attacks as those where attackers do not need to see the model itself but can infer its workings through inputs and outputs 5. This type of attack is becoming more common as models are deployed on devices like security cameras, which may not have the latest security patches.

The biggest problem is I go speak at security conferences and they're like, I don't know about this machine learning thing.

--- Katharine Jarmul

Justin Beyer concludes by stressing the importance of collaboration between security and machine learning communities to address these issues effectively 6.