OpenAI's Operator scored 38% on real computer tasks — and panics instead of recovering from errors
4.4/10
Severity
Logic Error
Failure Mode
Reproducible
No
Date
July 15, 2025
Expected Behavior
Complete common computer tasks reliably and recover gracefully when a step fails.
What Actually Happened
Six months after launch, OpenAI's Operator scored 38% on OSWorld, a benchmark of real computer tasks — meaning about two in five fail. On errors, computer-use agents often panic, double down on the wrong action, ignore error messages, and don't retry differently. Failures range from a typo in an email to buying the wrong item to permanently deleting a document.
Damage Assessment
At a 38% success rate, routine autonomous actions — sending the wrong email to a customer, purchasing the wrong item — are common outcomes, not edge cases, especially given poor error recovery.
Full Report
OpenAI launched Operator, its computer-using agent, in January 2025; six months later it scored 38% on OSWorld, a benchmark of real computer tasks — roughly two of every five tasks fail. Beyond the raw success rate, the more damaging finding is error recovery: when a step goes wrong, computer-use agents tend to panic, double down on the wrong action, ignore error messages, and fail to retry a different way. Because these agents act in the real world, the failure surface includes purchasing the wrong item, sending the wrong email to a customer, or permanently deleting a document. OpenAI's own mitigation — training the agent to ask for confirmation before finalizing side-effectful actions like orders and emails — is an admission that unsupervised autonomy at current reliability is unsafe for consequential tasks.
Incident Metadata
- Agent
- Openai Operator
- Failure Mode
- Logic Error
- Root Cause
- Confidence Miscalibration
- Task Type
- other
- Domain
- frontend
- Source
- benchmark