# OTelBench (OpenTelemetry instrumentation tasks) (OTelBench)

> Quesma's OTelBench asks coding agents to add OpenTelemetry tracing, metrics and logging to small existing services in 11 programming languages. The score is the share of runs where the service then emits correct telemetry.

Canonical page: https://benchlm.ai/benchmarks/otelbench

- Category: [Coding](/coding)
- Last updated: November 13, 2026

## About OTelBench

- Year: 2026
- Tasks: 23 OpenTelemetry instrumentation tasks in 11 languages, three attempts each (69 runs)
- Format: Agent edits a service in a Linux terminal, Harbor framework
- Difficulty: Distributed tracing and context propagation across services
- Paper: [Benchmarking OpenTelemetry: can AI trace your failed login? (Quesma)](https://quesma.com/blog/introducing-otel-bench/)

Each task gives the agent a short service or a group of microservices, around 300 lines of code, and a Linux terminal. The prompt asks for instrumentation that follows OpenTelemetry conventions, matches the business domain and sends data to the endpoint set by the standard environment variable. A test then drives the service and checks the telemetry that comes out, for example that two separate user actions produce two separate traces and that a trace survives the hop from one service to the next. There are 23 tasks across Go, Java, C++, Python, JavaScript, PHP, Ruby, Rust, Erlang, .NET and Swift, each attempted three times, so a model's score is out of 69 runs. The published run covered 14 models and 966 runs and cost $522 in tokens. Tasks run in the Harbor framework and are published.

OTelBench is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (13 models)

| Rank | Model | Creator | Score | Who ran it |
|------|-------|---------|-------|-----------|
| 1 | [Claude Opus 4.5](/models/claude-opus-4-5) | Anthropic | 29.0% | Run by OTelBench |
| 2 | [GPT-5.2](/models/gpt-5-2) | OpenAI | 26.1% | Run by OTelBench |
| 3 | [Claude Sonnet 4.5](/models/claude-sonnet-4-5) | Anthropic | 21.7% | Run by OTelBench |
| 4 | [Gemini 3 Flash](/models/gemini-3-flash) | Google | 18.8% | Run by OTelBench |
| 5 | [Gemini 3 Pro](/models/gemini-3-pro) | Google | 15.9% | Run by OTelBench |
| 6 | [GPT-5.2-Codex](/models/gpt-5-2-codex) | OpenAI | 15.9% | Run by OTelBench |
| 7 | [GPT-5.1](/models/gpt-5-1) | OpenAI | 14.5% | Run by OTelBench |
| 8 | [GLM-4.7](/models/glm-4-7) | Z.AI | 13.0% | Run by OTelBench |
| 9 | [DeepSeek V3.2](/models/deepseek-v3-2) | DeepSeek | 11.6% | Run by OTelBench |
| 10 | [GPT-5.1-Codex-Max](/models/gpt-5-1-codex-max) | OpenAI | 11.6% | Run by OTelBench |
| 11 | [Claude Haiku 4.5](/models/claude-haiku-4-5) | Anthropic | 5.8% | Run by OTelBench |
| 12 | [Grok 4](/models/grok-4) | xAI | 4.3% | Run by OTelBench |
| 13 | [Grok 4.1 Fast](/models/grok-4-1-fast) | xAI | 2.9% | Run by OTelBench |

## FAQ

### What does OTelBench measure?

Whether a coding agent can instrument an existing service with OpenTelemetry so that it emits correct traces, metrics and logs. The agent edits around 300 lines of real-looking service code in a terminal, and a test checks the telemetry the service produces afterwards. Tasks cover 11 programming languages.

### Which model leads OTelBench?

Claude Opus 4.5 leads the January 16, 2026 leaderboard with 20 of 69 runs passed, or 29.0%. GPT-5.2 follows with 18 runs and Claude Sonnet 4.5 with 15. Gemini 3 Flash passed 13 runs, two more than Gemini 3 Pro, at a fraction of the cost.

### Why are OTelBench scores so low?

Nine of the 23 tasks were never solved by any of the 14 models. A common failure was merging two separate user actions into a single trace, or losing the trace when a request moved from one service to another. Quesma describes the tasks as easy for a site reliability engineer.

## Compare Top Models on OTelBench

- [Claude Opus 4.5 vs GPT-5.2](/compare/claude-opus-4-5-vs-gpt-5-2)
- [GPT-5.2 vs Claude Sonnet 4.5](/compare/claude-sonnet-4-5-vs-gpt-5-2)
- [Claude Sonnet 4.5 vs Gemini 3 Flash](/compare/claude-sonnet-4-5-vs-gemini-3-flash)
- [Gemini 3 Flash vs Gemini 3 Pro](/compare/gemini-3-flash-vs-gemini-3-pro)