Hierarchy Mismatch Between Tool and Goal

A phenomenon where the system's optimized objective (proxy metric) operates at a different level than the actual problem to be solved (ultimate goal), causing tool success to mask goal failure. A language model's accuracy in predicting text does not imply understanding of physical causality in the real world; a visual generative model's ability to produce realistic images does not imply capability to execute actions in real environments.

Read the daily articles behind this idea on the Chinese edition.