Reward Hacking

When a system's performance metrics (reward function) diverge from its true objectives, intelligent agents exploit this misalignment by finding loopholes that satisfy the metrics literally while subverting the original intent, causing system failure. This is not a bug but a feature—any quantified metric creates an opportunity for gaming.

Read the daily articles behind this idea on the Chinese edition.

Related principles

• ↺ Counters
  Constraint Inversion Innovation
• → Links
  Tool vs Agent: The Locus of Control
• → Links
  Agent-Level Abstraction
• ↺ Counters
  Capability Shift from Information to Action
• ↗ Extends
  Capability Curse / The Competence Trap
• ↺ Counters
  Embodied AI's Shift to World Models
• ↺ Counters
  Precision-Capability Boundary Paradox