Goodheart's Curse

Sources:

Goodheart's Curse refers to the phenomenon where optimizing for a specific metric leads to a distortion of the original goal, resulting in unintended consequences. On The Lunar Society podcast, explains this by using the example of carpentry tests. Initially, those good at carpentry might score high on a test, but once people start optimizing to score well on the test itself, rather than truly improving their carpentry skills, the test scores no longer reflect actual carpentry ability 1.

Essentially, Goodheart's Curse occurs when the measure used for optimization becomes the target, causing a decoupling from the real objective. This distortion arises because the optimization pressure focuses on the metric rather than the broader goal it was supposed to measure.

For more details, you can listen to Eliezer Yudkowsky's discussion on The Lunar Society.

Feel free to ask if you need more information on related topics!

Optimization Pressure and Goodheart's Curse

Eliezer discusses how optimization pressure can lead to unintended consequences and the phenomenon known as Goodheart's Curse. He uses the example of test scores and carpentry to illustrate how optimizing for a specific metric can result in a loss of overall proficiency.

The Lunar Society

Eliezer Yudkowsky - Why AI Will Kill Us, Aligning LLMs, Nature of Intelligence, SciFi, & Rationality
1
RELATED QUESTIONS