STUPID-2026-0039

In independent testing, Devin completed just 3 of 20 real-world tasks (15%)

1.0low
January 15, 2025Source checkedStudy · not counted in agent averages

Why 1/10

A capability benchmark (3 of 20 tasks completed), not a damaging event.

Scope
0/4
Time, tokens or the agent's own working state only
Recoverability
0/3
Nothing was lost
Did it happen
1/2
Demonstrated (proof of concept, reproduction, controlled test or near-miss) with no real victim reported
Beyond the user
0/1
Only the user's own non-production work

The score is the sum of these four factors. How each factor is scored · scored 2026-10-29

  1. Instruction given

    Autonomously complete 20 assigned real-world engineering tasks.

  2. Expected behavior

    Complete tasks as marketed for an 'autonomous AI software engineer.'

  3. Actual behavior

    In an independent Answer.AI evaluation of 20 tasks, only 3 succeeded, 14 failed outright, and 3 were inconclusive — a ~15% real-world success rate, far below the impression left by curated benchmark demos.

  4. Damage

    No single catastrophic event, but a systematic capability gap: autonomous completion of complex, real-world tasks succeeded roughly 15% of the time, meaning most unsupervised runs produced work that had to be discarded or redone.

Full write-up

Marketed as 'the first AI software engineer,' Devin's autonomous real-world reliability looked very different from its demo reel. In an independent evaluation by Answer.AI, Devin was assigned 20 real tasks: only 3 succeeded, 14 failed outright, and 3 were inconclusive — about a 15% success rate. That tracks with its 13.86% score on SWE-Bench Verified at launch. The gap illustrates a systemic pattern with autonomous coding agents: impressive on curated, well-scoped benchmark tasks, but on messy real-world work most unsupervised runs produce output that must be discarded or heavily corrected. The failure here isn't one dramatic incident — it's the quiet, systematic unreliability that marketing obscures.

Classification

Agent
Devin
Failure mode
Other
Root cause
Confidence Miscalibration
Domain
Backend
MIT AI Risk Repository (our mapping)
7.3 Lack of capability or robustness taxonomy
Source
Benchmark

Related incidents

1.0
OpenAI's Operator scored 38% on real computer tasks — and panics instead of recovering from errors
4.0
Salesforce Agentforce hit a 77% B2B failure rate — and Salesforce admitted it was 'more confident than we should have been'
4.0
Uber burned its entire annual AI coding budget in ~4 months after rolling out Claude Code to 5,000 engineers
3.0
Cyera study: 344 verified enterprise agent-damage cases, 188 with no attacker involved
Discuss on Hacker News
Cite this record
StupidLLM. "STUPID-2026-0039: In independent testing, Devin completed just 3 of 20 real-world tasks (15%)." AI Agent Incident Database, 2026. https://www.stupidllm.com/incident/STUPID-2026-0039/ (accessed 2026-10-29). CC BY 4.0.

← STUPID-2026-0038By IDSTUPID-2026-0040 →

Get told when an agent breaks something

We document AI agent failures daily, severity-scored against a published scale. Each Monday, if any scored 7 or above that week — deleted data, leaked secrets, broken production — you get one email with the sources. When nothing did, you get nothing.

This database is callable over MCP — query it from inside your agent.