Root cause

Confidence Miscalibration

6 documented incidents where confidence miscalibration was the underlying cause, plus 7 studies listed separately.

What this cause produces

  • Other5
  • Hallucination3
  • Infinite Loop2
  • Destructive Action1
  • Ignored Instructions1
  • Logic Error1
7.0
Two AI agents ping-ponged for 11 days and ran up a $47,000 bill — neither noticed anything wrong
6.0
Claude Code ran rm -rf on duplicate photo folders before verifying an earlier move had actually worked, destroying ~2,969 irreplaceable personal photos (GitHub #92737)
5.0
An AI agent spun up duplicate CloudFormation stacks on every error and ran up a $6,531 AWS bill
5.0
AI 'slop' vulnerability reports flooded curl until it killed its bug bounty
5.0
Cursor's own support AI 'Sam' invented a one-device login policy, triggering subscription cancellations
2.0
Sakana's 'AI Scientist' rewrote its own code to bypass its timeout and looped endlessly calling itself

Studies and surveys

Benchmarks, surveys and patterns rather than single failures. Not counted above.

4.0
Salesforce Agentforce hit a 77% B2B failure rate — and Salesforce admitted it was 'more confident than we should have been'
4.0
Uber burned its entire annual AI coding budget in ~4 months after rolling out Claude Code to 5,000 engineers
4.0
AI 'CVE slop' is drowning open-source maintainers: 60-80% of HackerOne submissions now invalid
3.0
Cyera study: 344 verified enterprise agent-damage cases, 188 with no attacker involved
1.0
The runaway-cost pattern, quantified: agentic coding tools burn 10-100x more tokens and can rival developer pay
1.0
OpenAI's Operator scored 38% on real computer tasks — and panics instead of recovering from errors
1.0
In independent testing, Devin completed just 3 of 20 real-world tasks (15%)