THREAT OPS › Threat News › Why AI Needs a “Genie Coefficient”
Why AI Needs a “Genie Coefficient”
<p><em>This essay was written with Barath Raghavan, and originally appeared in <a href="https://spectrum.ieee.org/ai-agent-benchmark">IEEE Spectrum</a>.</em></p> <p>Major benchmarks measure what AI can do. None measure whether it does what you mean: the distance between what you ask an AI to do and the unspoken assumptions about how you want the AI to do it. We propose a new metric: the Genie coef
MITRE ATT&CK techniques
- Code RepositoriesT1593.003
- CredentialsT1589.001
- Code RepositoriesT1213.003
- Code RepositoriesAML.T0095.000
Indicators of compromise
- https://spectrum.ieee.org/ai-agent-benchmarkurl
- https://spectrum.ieee.org/tag/terry-winogradurl
- https://metarationality.com/purpose-of-meaningurl
- https://metarationality.com/reasonable-referenceurl
- https://spectrum.ieee.org/tag/agentic-aiurl
- https://spectrum.ieee.org/tag/alexaurl
- https://spectrum.ieee.org/tag/siriurl
- https://simonwillison.net/2026/Jun/11/fable-is-relentlessly-proactive/url
- https://ourworldindata.org/what-is-the-gini-coefficienturl
- https://www.fastly.com/blog/using-gini-coefficient-plan-edge-capacityurl
- https://spectrum.ieee.org/prompt-injection-attackurl
- https://www.cna.org/analyses/2022/09/goodharts-lawurl
- https://metr.org/blog/2026-06-26-gpt-5-6-sol/url
- https://www.lesswrong.com/posts/qJYMbrabcQqCZ7iqm/impossiblebench-measuring-reward-hacking-in-llm-coding-1url
- https://taubench.com/url
- https://spectrum.ieee.org/ai-agents-safetyurl
- https://www.law.cornell.edu/wex/mens_reaurl