OTelBench (OpenTelemetry instrumentation tasks) (OTelBench)
We show this table for reference; we do not rank on it.
Quesma's OTelBench asks coding agents to add OpenTelemetry tracing, metrics and logging to small existing services in 11 programming languages. The score is the share of runs where the service then emits correct telemetry.
Benchmark score on OTelBench — November 13, 2026
We compile the OTelBench rows from benchmark-owner or independent runs. Claude Opus 4.5 leads the table at 29.0%, followed by GPT-5.2 (26.1%) and Claude Sonnet 4.5 (21.7%). We do not use these results to rank models overall.
Claude Opus 4.5
Anthropic
Run by OTelBenchGPT-5.2
OpenAI
Run by OTelBenchClaude Sonnet 4.5
Anthropic
Run by OTelBench13 modelsCodingCurrentDisplay onlyUpdated November 13, 2026
Benchmark score table (13 models)
| Rank | Model / configuration | Score | Who ran it | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | Claude Opus 4.5AnthropicRun by OTelBench | 29.0% | Run by OTelBench | Not reported | Closed |
| 2 | GPT-5.2OpenAIRun by OTelBench | 26.1% | Run by OTelBench | Not reported | Closed |
| 3 | Claude Sonnet 4.5AnthropicRun by OTelBench | 21.7% | Run by OTelBench | Not reported | Closed |
| 4 | Gemini 3 FlashGoogleRun by OTelBench | 18.8% | Run by OTelBench | Not reported | Closed |
| 5 | GPT-5.2-CodexOpenAIRun by OTelBench | 15.9% | Run by OTelBench | Not reported | Closed |
| 6 | Gemini 3 ProGoogleRun by OTelBench | 15.9% | Run by OTelBench | Not reported | Closed |
| 7 | GPT-5.1OpenAIRun by OTelBench | 14.5% | Run by OTelBench | Not reported | Closed |
| 8 | GLM-4.7Z.AIRun by OTelBench | 13.0% | Run by OTelBench | Not reported | Open |
| 9 | GPT-5.1-Codex-MaxOpenAIRun by OTelBench | 11.6% | Run by OTelBench | Not reported | Closed |
| 10 | DeepSeek V3.2DeepSeekRun by OTelBench | 11.6% | Run by OTelBench | Not reported | Open |
| 11 | Claude Haiku 4.5AnthropicRun by OTelBench | 5.8% | Run by OTelBench | Not reported | Closed |
| 12 | Grok 4xAIRun by OTelBench | 4.3% | Run by OTelBench | Not reported | Closed |
| 13 | Grok 4.1 FastxAIRun by OTelBench | 2.9% | Run by OTelBench | Not reported | Closed |
- Lab-reported
- Benchmark-run
- Independent
How to read this leaderboard
Higher is better, and every model does badly. Claude Opus 4.5 leads with 20 of 69 runs (29.0%), ahead of GPT-5.2 with 18 and Claude Sonnet 4.5 with 15. One run is worth about 1.4 points. Nine of the 23 tasks were not solved by any model in any attempt, including all of the Java, Ruby, Swift and Erlang microservice tasks, so the score mostly reflects the easier C++, Go and Python tasks.
Operator receipt: 13 sourced rows are currently displayable on this page; the leading published row is Claude Opus 4.5 at 29.0%.
Honest limit: The leaderboard dates from January 16, 2026 and has not been rerun on models released since, so it shows where late-2025 models stood, not today's. The services are small. A pass means the checks on emitted telemetry succeeded, which is narrower than instrumentation an on-call engineer would be happy with. We show it as display-only; it does not enter overall or category rankings.
Among the reported OTelBench rows, Claude Opus 4.5 is first at 29.0%. The third row is 7.3 points behind. The broader top-10 range is 17.4 points, so the table still separates the published systems.
13 models have been evaluated on OTelBench. The benchmark falls in the Coding category. We keep these results separate from the weighted global scoring system, so they remain source-specific evidence. OTelBench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About OTelBench
Year
2026
Tasks
23 OpenTelemetry instrumentation tasks in 11 languages, three attempts each (69 runs)
Format
Agent edits a service in a Linux terminal, Harbor framework
Difficulty
Distributed tracing and context propagation across services
Each task gives the agent a short service or a group of microservices, around 300 lines of code, and a Linux terminal. The prompt asks for instrumentation that follows OpenTelemetry conventions, matches the business domain and sends data to the endpoint set by the standard environment variable. A test then drives the service and checks the telemetry that comes out, for example that two separate user actions produce two separate traces and that a trace survives the hop from one service to the next. There are 23 tasks across Go, Java, C++, Python, JavaScript, PHP, Ruby, Rust, Erlang, .NET and Swift, each attempted three times, so a model's score is out of 69 runs. The published run covered 14 models and 966 runs and cost $522 in tokens. Tasks run in the Harbor framework and are published.
Freshness and provenance
Version
OTelBench 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does OTelBench measure?
Whether a coding agent can instrument an existing service with OpenTelemetry so that it emits correct traces, metrics and logs. The agent edits around 300 lines of real-looking service code in a terminal, and a test checks the telemetry the service produces afterwards. Tasks cover 11 programming languages.
Which model leads OTelBench?
Claude Opus 4.5 leads the January 16, 2026 leaderboard with 20 of 69 runs passed, or 29.0%. GPT-5.2 follows with 18 runs and Claude Sonnet 4.5 with 15. Gemini 3 Flash passed 13 runs, two more than Gemini 3 Pro, at a fraction of the cost.
Why are OTelBench scores so low?
Nine of the 23 tasks were never solved by any of the 14 models. A common failure was merging two separate user actions into a single trace, or losing the trace when a request moved from one service to another. Quesma describes the tasks as easy for a site reliability engineer.
Compare top models on OTelBench
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.