News
AI Still Can't Beat the On-Call Engineer: Here's Why
TL;DR
ARFBench is the first AI benchmark based on real production incidents, with GPT-5 achieving 62.7% accuracy, while human experts reach 72.7%. A combined AI-human model could potentially achieve 87.2% accuracy, highlighting the current limitations of AI in replacing on-call engineers.
Key points
- ARFBench is the first AI benchmark built from real production incidents
- GPT-5 leads AI models with 62.7% accuracy
- Human experts achieve 72.7% accuracy in incident response
- A combined AI-human model could reach 87.2% accuracy
- Trillions of dollars are lost each year due to system outages
In brief
- ARFBench is the first AI benchmark built entirely from real production incidents.
- GPT-5 leads all existing AI models at 62.7% accuracy but falls short of domain experts at 72.7%.
- A theoretical model-expert oracle—combining AI and human judgment—hits 87.2% accuracy, setting the ceiling for what collaborative AI-human teams could achieve.
AI companies keep pitching autonomous site reliability engineer agents—AI that investigates production incidents in place of humans. Datadog ran the actual benchmark on real outages, and the best AI models can't yet beat the engineers they’re supposed to replace.
The benchmark is ARFBench (Anomaly Reasoning Framework Benchmark), a joint project from Datadog and Carnegie Mellon. Built from 63 real production incidents, extracted from engineers' own Slack threads during live emergencies—750 multiple-choice questions covering 142 monitoring metrics and 5.38 million data points, every question verified by hand. No synthetic data. No textbook scenarios.
"Trillions of dollars are lost each year due to system outages," the researchers write. The benchmark tests whether AI can actually help change that.
“Despite the central role of such question-driven analysis in incident response, it remains unclear whether modern foundation models can reliably answer the kinds of time series questions engineers ask in practice,” the paper reads.
Questions come in three tiers. Tier I: Does an anomaly exist in this chart? Tier II: When did it start, how severe is it, what type?
The Tier III—the hardest—requires cross-metric reasoning: Is this chart causing the problem in that other chart? That's where AI falls apart. GPT-5 scores just 47.5% F1 on Tier III questions, a metric that penalizes models for gaming answers by picking the most common class.
"Despite the central role of such question-driven analysis in incident response, it remains unclear whether modern foundation models can reliably answer the kinds of time series questions engineers ask in practice," the researchers write.
How every model stacked up
GPT-5 led all existing models at 62.7% accuracy—on a test where random guessing gets 24.5%. Gemini 3 Pro scored 58.1%. Claude Opus 4.6: 54.8%. Claude Sonnet 4.5: 47.2%.
Domain experts scored 72.7% accuracy. Non-domain experts—time series researchers at Datadog without extensive observability experience—still hit 69.7%.
No AI model beat either human baseline.
Q&A
What is ARFBench and how was it created?
ARFBench is an AI benchmark developed from 63 real production incidents, using data extracted from engineers' Slack threads during emergencies.
How accurate is GPT-5 compared to human engineers in incident response?
GPT-5 achieves 62.7% accuracy, while human engineers reach 72.7% accuracy in handling production incidents.
What is the potential accuracy of a combined AI-human model?
A theoretical AI-human collaborative model could achieve an accuracy of 87.2%, setting a new benchmark for incident response.
Why is it important to improve AI in incident response?
Improving AI in incident response is crucial as system outages cost trillions of dollars annually, and effective AI could help mitigate these losses.
