BinaryAudit (backdoor detection in compiled binaries) (BinaryAudit)
We show this table for reference; we do not rank on it.
Quesma's BinaryAudit asks AI agents to find backdoors and timebombs planted in compiled, stripped binaries of real server software, using the reverse engineering tools Ghidra and radare2. The score here is the share of detection runs that found the planted code.
Benchmark score on BinaryAudit — November 11, 2026
We compile the BinaryAudit rows from benchmark-owner or independent runs. Gemini 3.1 Pro leads the table at 49.1%, followed by Claude Opus 4.6 (49.1%) and GPT-5.2-Codex (45.6%). We do not use these results to rank models overall.
Gemini 3.1 Pro
Claude Opus 4.6
Anthropic
GPT-5.2-Codex
OpenAI
18 modelsAgenticCurrentDisplay onlyUpdated November 11, 2026
Benchmark score table (18 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Gemini 3.1 ProGoogle | 49.1% | Not reported | Closed |
| 2 | Claude Opus 4.6Anthropic | 49.1% | Not reported | Closed |
| 3 | GPT-5.2-CodexOpenAI | 45.6% | Not reported | Closed |
| 4 | Gemini 3 ProGoogle | 43.9% | Not reported | Closed |
| 5 | GPT-5.3 CodexOpenAI | 42.1% | Not reported | Closed |
| 6 | Claude Opus 4.5Anthropic | 36.8% | Not reported | Closed |
| 7 | Gemini 3 FlashGoogle | 36.8% | Not reported | Closed |
| 8 | Claude Sonnet 4.5Anthropic | 26.3% | Not reported | Closed |
| 9 | Kimi K2.5Moonshot AI | 24.6% | Not reported | Open |
| 10 | GLM-4.7Z.AI | 24.6% | Not reported | Open |
| 11 | Gemini 2.5 ProGoogle | 21.1% | Not reported | Closed |
| 12 | GLM-5Z.AI | 19.3% | Not reported | Open |
| 13 | GPT-5.4OpenAI | 17.5% | Not reported | Closed |
| 14 | GPT-5.2OpenAI | 17.5% | Not reported | Closed |
| 15 | DeepSeek V3.2DeepSeek | 12.3% | Not reported | Open |
| 16 | Claude Haiku 4.5Anthropic | 10.5% | Not reported | Closed |
| 17 | Grok 4xAI | 8.8% | Not reported | Closed |
| 18 | Grok 4.1 FastxAI | 7.0% | Not reported | Closed |
How to read this leaderboard
Higher is better, and the ceiling so far is low: Gemini 3.1 Pro and Claude Opus 4.6 lead with 28 of 57 detection runs (49.1%). One run is worth 1.8 points, so models two or three points apart are tied. Read the detection rate next to the false alarms in the row note. Gemini 3 Pro found the backdoor in 25 of 57 runs but also flagged 15 of 23 clean runs, while GPT-5.2-Codex at xhigh found 26 and flagged none of 24.
Operator receipt: 18 sourced rows are currently displayable on this page; the leading published row is Gemini 3.1 Pro at 49.1%.
Honest limit: The backdoors are planted by the benchmark's authors and are, in their words, not particularly sophisticated, so a pass does not show a model would catch a real supply-chain attack. The leaderboard was last updated on March 7, 2026 and covers models released up to then. OpenAI models appear at several reasoning efforts; we store the highest effort listed and name the others in the row note. We show it as display-only; it does not enter overall or category rankings.
Among the reported BinaryAudit rows, Gemini 3.1 Pro is first at 49.1%. The third row is 3.5 points behind. The broader top-10 range is 24.5 points, so the table still separates the published systems.
18 models have been evaluated on BinaryAudit. The benchmark falls in the Agentic category. We keep these results separate from the weighted global scoring system, so they remain source-specific evidence. BinaryAudit is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About BinaryAudit
Year
2026
Tasks
19 backdoor and timebomb detection tasks on stripped binaries, three runs each (57 runs); 6 tooling and 8 clean-binary tasks reported alongside
Format
Agent with Ghidra and radare2 in a terminal, Harbor framework
Difficulty
Reverse engineering without source code or symbols
Quesma built the tasks with Michał "Redford" Kowalczyk of Dragon Sector. They took open-source network software (the lighttpd web server, the dnsmasq DNS server, the Dropbear SSH server and the Sozu load balancer among them), added a backdoor or timebomb by hand, and compiled it to a stripped executable with no symbols or source. The agent has to say whether the binary is malicious and where. There are 33 tasks in three groups: 19 detection tasks, 6 tooling tasks that only ask for a decompilation with Ghidra or radare2, and 8 clean binaries where the right answer is that nothing is wrong. Each task is run three times, so the detection score is out of 57 runs. Agents are limited to open-source tools. Tasks run in the Harbor framework and are published. Each row note also carries the model's tooling result and how often it raised a false alarm on a clean binary.
Freshness and provenance
Version
BinaryAudit 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does BinaryAudit measure?
Whether an AI agent can tell that a compiled program has been tampered with. Each detection task hands the agent a stripped binary of real server software with a planted backdoor or timebomb, plus Ghidra and radare2, and asks whether it is malicious and where the malicious code sits.
Which model leads BinaryAudit?
Gemini 3.1 Pro and Claude Opus 4.6 tie at 28 of 57 detection runs, or 49.1%, on the March 7, 2026 leaderboard. GPT-5.2-Codex at xhigh effort follows with 26. Claude Opus 4.6 raised a false alarm on 5 of 23 clean runs, and Gemini 3.1 Pro on 4 of 24.
Why does the false alarm rate matter?
A scanner that flags everything catches every backdoor and is useless. BinaryAudit includes eight clean binaries for this reason. Gemini 2.5 Pro reported a backdoor in 18 of 21 clean runs, so its detection result means less than the same number from a model that flagged none.
Compare top models on BinaryAudit
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.