# BinaryAudit (backdoor detection in compiled binaries) (BinaryAudit)

> Quesma's BinaryAudit asks AI agents to find backdoors and timebombs planted in compiled, stripped binaries of real server software, using the reverse engineering tools Ghidra and radare2. The score here is the share of detection runs that found the planted code.

Canonical page: https://benchlm.ai/benchmarks/binaryaudit

- Category: [Agentic](/agentic)
- Last updated: November 11, 2026

## About BinaryAudit

- Year: 2026
- Tasks: 19 backdoor and timebomb detection tasks on stripped binaries, three runs each (57 runs); 6 tooling and 8 clean-binary tasks reported alongside
- Format: Agent with Ghidra and radare2 in a terminal, Harbor framework
- Difficulty: Reverse engineering without source code or symbols
- Paper: [We hid backdoors in ~40 MB binaries and asked AI + Ghidra to find them (Quesma)](https://quesma.com/blog/introducing-binaryaudit/)

Quesma built the tasks with Michał "Redford" Kowalczyk of Dragon Sector. They took open-source network software (the lighttpd web server, the dnsmasq DNS server, the Dropbear SSH server and the Sozu load balancer among them), added a backdoor or timebomb by hand, and compiled it to a stripped executable with no symbols or source. The agent has to say whether the binary is malicious and where. There are 33 tasks in three groups: 19 detection tasks, 6 tooling tasks that only ask for a decompilation with Ghidra or radare2, and 8 clean binaries where the right answer is that nothing is wrong. Each task is run three times, so the detection score is out of 57 runs. Agents are limited to open-source tools. Tasks run in the Harbor framework and are published. Each row note also carries the model's tooling result and how often it raised a false alarm on a clean binary.

BinaryAudit is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (18 models)

| Rank | Model | Creator | Score |
|------|-------|---------|-------|
| 1 | [Gemini 3.1 Pro](/models/gemini-3-1-pro) | Google | 49.1% |
| 2 | [Claude Opus 4.6](/models/claude-opus-4-6) | Anthropic | 49.1% |
| 3 | [GPT-5.2-Codex](/models/gpt-5-2-codex) | OpenAI | 45.6% |
| 4 | [Gemini 3 Pro](/models/gemini-3-pro) | Google | 43.9% |
| 5 | [GPT-5.3 Codex](/models/gpt-5-3-codex) | OpenAI | 42.1% |
| 6 | [Gemini 3 Flash](/models/gemini-3-flash) | Google | 36.8% |
| 7 | [Claude Opus 4.5](/models/claude-opus-4-5) | Anthropic | 36.8% |
| 8 | [Claude Sonnet 4.5](/models/claude-sonnet-4-5) | Anthropic | 26.3% |
| 9 | [Kimi K2.5](/models/kimi-k2-5) | Moonshot AI | 24.6% |
| 10 | [GLM-4.7](/models/glm-4-7) | Z.AI | 24.6% |
| 11 | [Gemini 2.5 Pro](/models/gemini-2-5-pro) | Google | 21.1% |
| 12 | [GLM-5](/models/glm-5) | Z.AI | 19.3% |
| 13 | [GPT-5.4](/models/gpt-5-4) | OpenAI | 17.5% |
| 14 | [GPT-5.2](/models/gpt-5-2) | OpenAI | 17.5% |
| 15 | [DeepSeek V3.2](/models/deepseek-v3-2) | DeepSeek | 12.3% |
| 16 | [Claude Haiku 4.5](/models/claude-haiku-4-5) | Anthropic | 10.5% |
| 17 | [Grok 4](/models/grok-4) | xAI | 8.8% |
| 18 | [Grok 4.1 Fast](/models/grok-4-1-fast) | xAI | 7.0% |

## FAQ

### What does BinaryAudit measure?

Whether an AI agent can tell that a compiled program has been tampered with. Each detection task hands the agent a stripped binary of real server software with a planted backdoor or timebomb, plus Ghidra and radare2, and asks whether it is malicious and where the malicious code sits.

### Which model leads BinaryAudit?

Gemini 3.1 Pro and Claude Opus 4.6 tie at 28 of 57 detection runs, or 49.1%, on the March 7, 2026 leaderboard. GPT-5.2-Codex at xhigh effort follows with 26. Claude Opus 4.6 raised a false alarm on 5 of 23 clean runs, and Gemini 3.1 Pro on 4 of 24.

### Why does the false alarm rate matter?

A scanner that flags everything catches every backdoor and is useless. BinaryAudit includes eight clean binaries for this reason. Gemini 2.5 Pro reported a backdoor in 18 of 21 clean runs, so its detection result means less than the same number from a model that flagged none.

## Compare Top Models on BinaryAudit

- [Gemini 3.1 Pro vs Claude Opus 4.6](/compare/claude-opus-4-6-vs-gemini-3-1-pro)
- [Claude Opus 4.6 vs GPT-5.2-Codex](/compare/claude-opus-4-6-vs-gpt-5-2-codex)
- [GPT-5.2-Codex vs Gemini 3 Pro](/compare/gemini-3-pro-vs-gpt-5-2-codex)
- [Gemini 3 Pro vs GPT-5.3 Codex](/compare/gemini-3-pro-vs-gpt-5-3-codex)