690: How to Catch and Fix Harmful Generative A.I. Outputs — with Krishna Gade

Inaccuracies

Generative AI systems often produce inaccurate outputs, posing challenges for trust and reliability. Krishna Gade explains that large language models (LLMs) can generate inconsistent responses when faced with counterfactual prompts, which are slight variations of the original question. This inconsistency is problematic, especially in enterprise settings where accuracy is crucial. Gade emphasizes the importance of tools like Fiddler Auditor to assess the robustness of these models by probing them with counterfactual questions to identify failure scenarios 1.

So these counterfactuals are slight iterations on a prompt to see how often the answer breaks down. So you get the sense of how robust it is.

--- Jon Krohn

This approach helps build a scorecard that developers can use to evaluate the reliability of AI outputs before deployment 2.

   

Bias & Privacy

Bias and privacy risks are significant concerns in AI outputs, as LLMs can inadvertently expose private data or exhibit bias against protected groups. Krishna Gade highlights that these issues can lead to harmful outputs, necessitating robust detection and mitigation strategies. The Fiddler Auditor, an open-source tool developed by Gade's team, offers solutions to identify and address these undesirable outputs, enhancing trust in AI systems 3.

They can be biased against protected groups, and they're susceptible to exposing private data.

--- Jon Krohn

By employing such tools, organizations can ensure their AI systems are safer and more reliable.

   

Governance

Effective governance and risk management are essential for maintaining AI system safety and compliance. Krishna Gade discusses the role of feedback controls and classifiers in detecting unsafe or biased content in AI responses. These mechanisms are crucial for creating comprehensive model risk management reports, which document potential failure scenarios and interactions within the model 4.

Essentially, it will have, hey, we've tested it against these thousand prompts. This is the overall robustness score.

--- Krishna Gade

Such reports are vital for regulated industries, helping them navigate the complexities of AI adoption while ensuring compliance with regulatory standards.

   

Compliance

In regulated industries, the adoption of generative AI requires navigating complex regulatory landscapes. Krishna Gade explains that tools like Fiddler Auditor assist companies in creating AI validation and governance reports, which are crucial for compliance. These reports provide insights into the robustness and safety of AI models, helping organizations make informed decisions about deploying AI technologies 4.

So this is like audit report that you can then share it with your MRM team and sort of get, and get them satisfied.

--- Krishna Gade

By ensuring transparency and accountability, such tools facilitate the responsible use of AI in sensitive sectors.