Methodology
StupidLLM uses a CVSS-inspired severity scoring system to rate every documented AI agent failure on a 0-10 scale. Each incident links to a source and is categorized by failure mode and root cause. STUPID-IDs and these scores are our own; they are not CVE identifiers or official CVSS assessments.
Severity Scoring (0-10)
Every score is the sum of four factors, and each incident page shows its breakdown under “Why this score”. For a security flaw, scope and recoverability describe what the documented exploit would do; “did it happen” then records whether anyone was actually harmed. Where a source does not say whether lost data was recovered, recoverability is scored 2, not 3.
- 0Time, tokens or the agent's own working state only
- 1A few files, one session, one credential, or a small cost
- 2One project, repository, server or service, or a cost above roughly $10,000
- 3A whole machine or drive, an organisation's production system or data, or many credentials
- 4Many organisations or the public: a flaw in a widely used product, mass exposure, a large outage
- 0Nothing was lost
- 1Fully recovered, with effort
- 2Partly lost, or loss reported without saying whether it was recovered
- 3Permanently lost or irreversible, including data that was exposed
- 0A study, benchmark or pattern, not a single event
- 1Demonstrated (proof of concept, reproduction, controlled test or near-miss) with no real victim reported
- 2Happened in real use
- 0Only the user's own non-production work
- 1Production systems, customers or third parties, or credentials and sensitive personal data
The total falls into one of these bands. Every incident then in the database (138) was re-scored against the factors on 29 October 2026; before that, scores were single judgments against the band descriptions alone, and 25 incidents sat at 10/10. Now a 10 needs harm that reached many people, could not be undone, and actually happened.
Causes irreversible damage: data loss, security breach, production outage.
Significant impact: corrupted codebase, API key exposure, system instability.
Moderate impact: logic errors, wasted resources, incorrect but recoverable output.
Minor issues: cosmetic bugs, harmless hallucinations, non-production impact.
Cross-references and guardrails
Incidents carry codes from the OWASP Top 10 for Agentic Applications (2026) and the MIT AI Risk Repository domain taxonomy, so they can be filed in an existing risk register. The names are copied from OWASP’s published list and MIT’s paper; the mapping of each incident is ours, not a classification by OWASP or MIT. Where nothing fits, an incident carries no code rather than a forced one: OWASP’s list covers agentic attack and misuse classes, so insecure code an agent wrote into an app has no ASI entry.
Incidents caused by a command a pattern can recognise also carry guardrail rules for Claude Code, Codex, Gemini CLI and Copilot CLI, written in each tool’s documented format. Rules appear only where they would have intercepted the command as reported.
Verification
Every incident links to a source (GitHub issue or PR, post, vendor report, news article). A source is evidence of what was reported, not always a primary account. Each record carries two labels, and they mean exactly this: Source checked — we read the linked source and it supports the record; Repro reported— the source says the failure reproduces or gives steps (we have not reproduced it ourselves). Records without a check say “source not yet checked” and are updated when we check them. We don't currently mark vendor confirmation or independent reproduction; when a vendor acknowledges an issue, the record cites it.
Source taxonomy
Every incident cites a source type, and the type informs how much weight the incident carries independent of its verification status. Direct artifacts (a pull request, a benchmark run) are trusted more than secondhand narration, and single first-person reports are weighted lowest until corroborated. Current counts across the 138-incident corpus:
Weighted case-by-case against the evidence it provides.
Moderate trust — reviewed by a publication, but relies on the outlet's own sourcing.
Highest trust — a pull request or commit is the failure itself, not a description of it.
Highest trust — published by researchers with disclosure obligations and technical detail to verify.
High trust — produced by a standardized, repeatable evaluation rather than a single anecdote.
Lower trust on its own — a first-person account without independent corroboration; verified before being treated as confirmed.
Moderate trust — first-person account, weighed on the specificity of evidence given.
Moderate trust — public discussion thread, weighed against corroborating detail.
Weighted case-by-case against the evidence it provides.
Sample size and ranking
An agent must have at least 3documented incidents before it appears in any ranked list. Mean severity computed over a single report is not a measurement, and presenting it as one would be misleading. Agents below that threshold are published and browsable, but they are shown in a separate “insufficient data” tier.
Entries such as multiple-agents, multiple-llms, and unknown-agent describe multi-agent or unattributed failures. They are collection buckets, not products, and are never counted toward any vendor.
Known limitations
- Reporting bias. Incidents are collected from public reports. Widely-used agents attract more scrutiny, so incident counts reflect attention as well as reliability and must not be read as a defect rate.
- No exposure denominator. We do not know how many total tasks each agent ran, so these counts cannot be converted into a failure probability.
- Long tail. Most tracked agents have only one or two documented incidents, which is why the ranking threshold exists.
- Severity is a judgment. Each factor is a reviewer’s reading of the source against the definitions above. The arithmetic is fixed; the factor values are judgment, shown on every incident so they can be challenged.