BullshitBench v2
We show this table for reference; we do not rank on it.
A benchmark that tests whether AI models challenge nonsensical, ill-posed, or logically flawed prompts instead of confidently generating incorrect answers. Measures the critical ability to push back on bad input.
Clear pushback rate on BullshitBench v2 — September 29, 2026 at 8:42 PM UTC
We mirror the published clear pushback rate view for BullshitBench v2. Claude Opus 4.8 leads the public snapshot at 95%, followed by Qwen3.8 Max (95%) and Claude Opus 4.8 (94%). We do not use these results to rank models overall.
Claude Opus 4.8
Anthropic
Qwen3.8 Max
Alibaba
Claude Opus 4.8
Anthropic
228 modelsReasoningCurrentDisplay onlyUpdated September 29, 2026 at 8:42 PM UTC
Clear pushback rate table (228 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Claude Opus 4.8Anthropic | 95% | Not reported | Closed |
| 2 | Qwen3.8 MaxAlibaba | 95% | Not reported | Open |
| 3 | Claude Opus 4.8Anthropic | 94% | Not reported | Closed |
| 4 | Claude Sonnet 4.6Anthropic | 91% | Not reported | Closed |
| 5 | Claude Opus 4.5Anthropic | 90% | Not reported | Closed |
| 6 | Claude Sonnet 4.6Anthropic | 89% | Not reported | Closed |
| 7 | Claude Opus 4.6Anthropic | 87% | Not reported | Closed |
| 8 | Claude Opus 4.6Anthropic | 83% | Not reported | Closed |
| 9 | Claude Opus 4.7Anthropic | 83% | Not reported | Closed |
| 10 | Claude Sonnet 5Anthropic | 80% | Not reported | Closed |
| 11 | Claude Sonnet 4.5Anthropic | 79% | Not reported | Closed |
| 12 | Claude Opus 4.5Anthropic | 79% | Not reported | Closed |
| 13 | Qwen3.8-27BAlibaba | 79% | 27B | Open |
| 14 | Claude Sonnet 5Anthropic | 78% | Not reported | Closed |
| 15 | Qwen3.5 397B (Reasoning)Alibaba | 78% | 397B (17B active) | Open |
| 16 | Claude Fable 5.1Anthropic | 77% | Not reported | Closed |
| 17 | Claude Haiku 4.5Anthropic | 77% | Not reported | Closed |
| 18 | Kimi K3Moonshot AI | 74% | Not reported | Pending |
| 19 | Claude Opus 4.7Anthropic | 74% | Not reported | Closed |
| 20 | Claude Sonnet 4.5Anthropic | 74% | Not reported | Closed |
| 21 | Claude Opus 5Anthropic | 73% | Not reported | Closed |
| 22 | Kimi K3Moonshot AI | 72% | Not reported | Pending |
| 23 | GLM-5.3Z.AI | 72% | Not reported | Open |
| 24 | Qwen3.7 MaxAlibaba | 72% | Not reported | Closed |
| 25 | Qwen3.6 PlusAlibaba | 72% | Not reported | Closed |
| 26 | Claude Haiku 4.5Anthropic | 71% | Not reported | Closed |
| 27 | Claude Opus 5Anthropic | 70% | Not reported | Closed |
| 28 | GPT-6 AstraOpenAI | 69% | Not reported | Closed |
| 29 | Qwen3.5 397BAlibaba | 69% | 397B (17B active) | Open |
| 30 | Grok 4.20 Multi-agentxAI | 67% | Not reported | Closed |
| 31 | GLM-5.3Z.AI | 66% | Not reported | Open |
| 32 | Gemini 3.5 Flash-LiteGoogle | 66% | Not reported | Closed |
| 33 | Grok 4.6xAI | 66% | Not reported | Closed |
| 34 | GPT-6.1 SolOpenAI | 65% | Not reported | Closed |
| 35 | Kimi K2.6Moonshot AI | 65% | 1,000B (32B active) | Open |
| 36 | stealth/ox-alpha (low)Stealth | 65% | Not reported | Unknown |
| 37 | Claude Fable 5.1Anthropic | 64% | Not reported | Closed |
| 38 | GPT-6 AstraOpenAI | 64% | Not reported | Closed |
| 39 | MiniMax M3MiniMax | 64% | Not reported | Open |
| 40 | Grok 4.20 Multi-agentxAI | 64% | Not reported | Closed |
| 41 | Claude Opus 5.5Anthropic | 63% | Not reported | Closed |
| 42 | Qwen3.6 PlusAlibaba | 63% | Not reported | Closed |
| 43 | MiniMax M3MiniMax | 63% | Not reported | Open |
| 44 | Claude Opus 5.5Anthropic | 62% | Not reported | Closed |
| 45 | MiMo-V2.5-ProXiaomi | 62% | 1,000B (42B active) | Closed |
| 46 | Claude Sonnet 5.5Anthropic | 60% | Not reported | Closed |
| 47 | Gemini 3.5 Flash-LiteGoogle | 60% | Not reported | Closed |
| 48 | GPT-6.1 SolOpenAI | 59% | Not reported | Closed |
| 49 | openai/gpt-6-luna-pro (max)OpenAI | 59% | Not reported | Unknown |
| 50 | Qwen3.6 PlusAlibaba | 59% | Not reported | Closed |
| 51 | GPT-6 SolOpenAI | 59% | Not reported | Closed |
| 52 | GPT-6 LunaOpenAI | 59% | Not reported | Closed |
| 53 | Claude Sonnet 5.5Anthropic | 58% | Not reported | Closed |
| 54 | GPT-6 LunaOpenAI | 58% | Not reported | Closed |
| 55 | openai/gpt-6-sol-pro (max)OpenAI | 57% | Not reported | Unknown |
| 56 | DeepSeek V4.1 FlashDeepSeek | 57% | 552B (16B active) | Open |
| 57 | DeepSeek V4.1 FlashDeepSeek | 57% | 552B (16B active) | Open |
| 58 | openai/gpt-6-luna-pro (none)OpenAI | 57% | Not reported | Unknown |
| 59 | Qwen3.7 MaxAlibaba | 57% | Not reported | Closed |
| 60 | Claude Fable 5Anthropic | 56% | Not reported | Closed |
| 61 | Grok 4.6xAI | 56% | Not reported | Closed |
| 62 | Grok 4.20xAI | 56% | Not reported | Closed |
| 63 | openai/gpt-6-sol-pro (none)OpenAI | 55% | Not reported | Unknown |
| 64 | Grok 4.5xAI | 55% | Not reported | Closed |
| 65 | GPT-6 SolOpenAI | 54% | Not reported | Closed |
| 66 | stealth/ox-alpha (max)Stealth | 54% | Not reported | Unknown |
| 67 | GPT-5.6 TerraOpenAI | 54% | Not reported | Closed |
| 68 | Grok 4.5xAI | 54% | Not reported | Closed |
| 69 | Grok 4.20xAI | 54% | Not reported | Closed |
| 70 | Nemotron 3 Super 120B A12BNVIDIA | 54% | 120.6B (12.7B active) | Open |
| 71 | Kimi K2.5Moonshot AI | 52% | 1,000B (32B active) | Open |
| 72 | Grok 4.3xAI | 50% | Not reported | Closed |
| 73 | Kimi K2.6Moonshot AI | 50% | 1,000B (32B active) | Open |
| 74 | Muse Spark 1.2Meta | 50% | Not reported | Closed |
| 75 | anthropic/claude-3.5-haikuAnthropic | 50% | Not reported | Unknown |
| 76 | Muse Spark 1.2Meta | 49% | Not reported | Closed |
| 77 | anthropic/claude-3.7-sonnet:thinkingAnthropic | 49% | Not reported | Unknown |
| 78 | nvidia/nemotron-3-ultra-550b-a55b (none)NVIDIA | 49% | 550B (55B active) | Unknown |
| 79 | GPT-5.6 SolOpenAI | 48% | Not reported | Closed |
| 80 | GPT-5.4OpenAI | 48% | Not reported | Closed |
| 81 | Gemini 3 ProGoogle | 48% | Not reported | Closed |
| 82 | GPT-5.6 SolOpenAI | 47% | Not reported | Closed |
| 83 | GPT-5.5OpenAI | 47% | Not reported | Closed |
| 84 | GPT-5.6 TerraOpenAI | 47% | Not reported | Closed |
| 85 | Nemotron 3 Super 120B A12BNVIDIA | 47% | 120.6B (12.7B active) | Open |
| 86 | Qwen3.6 PlusAlibaba | 46% | Not reported | Closed |
| 87 | Grok 4.3xAI | 46% | Not reported | Closed |
| 88 | Claude 3.5 SonnetAnthropic | 46% | Not reported | Closed |
| 89 | Claude Fable 5Anthropic | 45% | Not reported | Closed |
| 90 | GPT-5.5OpenAI | 45% | Not reported | Closed |
| 91 | GPT-5.5OpenAI | 45% | Not reported | Closed |
| 92 | GPT-5.2-CodexOpenAI | 45% | Not reported | Closed |
| 93 | GPT-5.1OpenAI | 45% | Not reported | Closed |
| 94 | anthropic/claude-3.7-sonnetAnthropic | 44% | Not reported | Unknown |
| 95 | Claude 4.1 OpusAnthropic | 43% | Not reported | Closed |
| 96 | Nemotron 3 Super 120B A12BNVIDIA | 43% | 120.6B (12.7B active) | Open |
| 97 | openrouter/hunter-alpha (none)Stealth | 43% | 1,000B | Unknown |
| 98 | GPT-5.4OpenAI | 42% | Not reported | Closed |
| 99 | Claude 4.1 OpusAnthropic | 42% | Not reported | Closed |
| 100 | GPT-5.6 LunaOpenAI | 41% | Not reported | Closed |
| 101 | GPT-5.3 InstantOpenAI | 40% | Not reported | Closed |
| 102 | nvidia/nemotron-3-ultra-550b-a55b (xhigh)NVIDIA | 40% | 550B (55B active) | Unknown |
| 103 | DeepSeek V4 Flash 0731DeepSeek | 39% | 284B (13B active) | Open |
| 104 | GPT-5 CodexCreator not reported | 39% | Not reported | Unknown |
| 105 | GPT-5.2-CodexOpenAI | 39% | Not reported | Closed |
| 106 | Gemini 3.6 FlashGoogle | 39% | Not reported | Closed |
| 107 | GPT-5.2OpenAI | 38% | Not reported | Closed |
| 108 | MiMo-V2.5-ProXiaomi | 38% | 1,000B (42B active) | Closed |
| 109 | GPT-5.6 LunaOpenAI | 37% | Not reported | Closed |
| 110 | Gemini 3.1 ProGoogle | 37% | Not reported | Closed |
| 111 | GPT-5.2-CodexOpenAI | 37% | Not reported | Closed |
| 112 | openrouter/healer-alpha (none)Stealth | 37% | Not reported | Unknown |
| 113 | GPT-5.5 ProOpenAI | 36% | Not reported | Closed |
| 114 | Gemini 3 Pro Deep ThinkGoogle | 36% | Not reported | Closed |
| 115 | DeepSeek V4 Pro 0813DeepSeek | 35% | 1,600B (49B active) | Open |
| 116 | Gemini 3.7 FlashGoogle | 35% | Not reported | Closed |
| 117 | MiMo-V2.5Xiaomi | 35% | Not reported | Closed |
| 118 | openrouter/hunter-alpha (xhigh)Stealth | 35% | 1,000B | Unknown |
| 119 | GPT-5.5 ProOpenAI | 34% | Not reported | Closed |
| 120 | GPT-5.5OpenAI | 34% | Not reported | Closed |
| 121 | Claude Opus 4Creator not reported | 34% | Not reported | Unknown |
| 122 | GPT-5.4 miniOpenAI | 32% | Not reported | Closed |
| 123 | GPT-5.4 miniOpenAI | 32% | Not reported | Closed |
| 124 | DeepSeek V4 Pro 0813DeepSeek | 32% | 1,600B (49B active) | Open |
| 125 | GPT-5.1-Codex-MaxOpenAI | 32% | Not reported | Closed |
| 126 | GPT-5.4 miniOpenAI | 31% | Not reported | Closed |
| 127 | Kimi K2.5 (Reasoning)Moonshot AI | 31% | 1,000B (32B active) | Closed |
| 128 | Gemini 3.1 ProGoogle | 31% | Not reported | Closed |
| 129 | Gemini 3.7 FlashGoogle | 31% | Not reported | Closed |
| 130 | GLM-5-TurboZ.AI | 31% | Not reported | Closed |
| 131 | Nemotron 3 Super 120B A12BNVIDIA | 31% | 120.6B (12.7B active) | Open |
| 132 | GLM-5.2Z.AI | 31% | 753B | Open |
| 133 | Claude 4 SonnetAnthropic | 30% | Not reported | Closed |
| 134 | Qwen3.8-27BAlibaba | 29% | 27B | Open |
| 135 | Claude 4 SonnetAnthropic | 29% | Not reported | Closed |
| 136 | Llama 4 MaverickMeta | 29% | 400B (17B active) | Open |
| 137 | GPT-5.2OpenAI | 28% | Not reported | Closed |
| 138 | Gemini 3.6 FlashGoogle | 28% | Not reported | Closed |
| 139 | GLM-5 (Reasoning)Z.AI | 28% | 744B (40B active) | Open |
| 140 | Nemotron 3 Nano 30B A3B (none)Creator not reported | 28% | 30B (3.5B active) | Unknown |
| 141 | Qwen3.8 MaxAlibaba | 27% | Not reported | Open |
| 142 | GPT-5.2 InstantOpenAI | 27% | Not reported | Closed |
| 143 | o3OpenAI | 26% | Not reported | Closed |
| 144 | openrouter/healer-alpha (xhigh)Stealth | 26% | Not reported | Unknown |
| 145 | GPT-5.1OpenAI | 25% | Not reported | Closed |
| 146 | Gemma 4 31BGoogle | 25% | 30.7B | Open |
| 147 | GPT-5.3 CodexOpenAI | 24% | Not reported | Closed |
| 148 | MiMo-V2.5Xiaomi | 24% | Not reported | Closed |
| 149 | GLM-5-TurboZ.AI | 23% | Not reported | Closed |
| 150 | GLM-5.1Z.AI | 22% | 754B | Open |
| 151 | Step 3.5 FlashStepFun | 22% | 196B (11B active) | Open |
| 152 | Laguna S 2.1Poolside | 22% | 118B (8B active) | Open |
| 153 | GPT-5Creator not reported | 21% | Not reported | Unknown |
| 154 | Gemma 4 26B A4BGoogle | 21% | 25.2B (3.8B active) | Open |
| 155 | Laguna S 2.1Poolside | 21% | 118B (8B active) | Open |
| 156 | GPT-5.3 CodexOpenAI | 20% | Not reported | Closed |
| 157 | DeepSeek V4 Flash 0731DeepSeek | 20% | 284B (13B active) | Open |
| 158 | Qwen3 Coder 480B A35BCreator not reported | 20% | 480B (35B active) | Unknown |
| 159 | Gemini 2.5 ProGoogle | 20% | Not reported | Closed |
| 160 | GLM-5Z.AI | 20% | 744B (40B active) | Open |
| 161 | Llama 4 ScoutMeta | 20% | 109B (17B active) | Open |
| 162 | Gemma 4 31BGoogle | 20% | 30.7B | Open |
| 163 | Gemini 3.5 FlashGoogle | 20% | Not reported | Closed |
| 164 | GPT-5.3 CodexOpenAI | 19% | Not reported | Closed |
| 165 | Grok 4.1 FastxAI | 19% | Not reported | Closed |
| 166 | Gemini 2.5 FlashGoogle | 19% | Not reported | Closed |
| 167 | Gemini 3.5 FlashGoogle | 19% | Not reported | Closed |
| 168 | GPT-5Creator not reported | 18% | Not reported | Unknown |
| 169 | DeepSeek V4 Flash 0731DeepSeek | 18% | 284B (13B active) | Open |
| 170 | GLM-5.1Z.AI | 18% | 754B | Open |
| 171 | GLM-5.2Z.AI | 17% | 753B | Open |
| 172 | Trinity-Large-ThinkingArcee AI | 17% | 398B (13B active) | Open |
| 173 | MiMo-V2-FlashXiaomi | 16% | 309B (15B active) | Open |
| 174 | Hy3Tencent | 16% | 295B (21B active) | Open |
| 175 | google/gemini-2.0-flash-001Google | 15% | Not reported | Unknown |
| 176 | DeepSeek V4 Pro 0813DeepSeek | 14% | 1,600B (49B active) | Open |
| 177 | meta-llama/llama-3.1-8b-instructMeta | 14% | 8B | Unknown |
| 178 | GPT-5.4 nanoOpenAI | 14% | Not reported | Closed |
| 179 | GPT-5.4 nanoOpenAI | 14% | Not reported | Closed |
| 180 | DeepSeek V4 Pro 0813DeepSeek | 14% | 1,600B (49B active) | Open |
| 181 | GPT-4.1OpenAI | 14% | Not reported | Closed |
| 182 | DeepSeek V4 Flash 0731DeepSeek | 14% | 284B (13B active) | Open |
| 183 | DeepSeek V3.2 (Thinking)DeepSeek | 13% | 685B (37B active) | Open |
| 184 | Step 3.5 FlashStepFun | 13% | 196B (11B active) | Open |
| 185 | Trinity-Large-ThinkingArcee AI | 13% | 398B (13B active) | Open |
| 186 | MiMo-V2-FlashXiaomi | 13% | 309B (15B active) | Open |
| 187 | GPT-4oOpenAI | 12% | Not reported | Closed |
| 188 | GPT-OSS 120BOpenAI | 12% | 117B (5.1B active) | Open |
| 189 | Gemma 4 26B A4BGoogle | 11% | 25.2B (3.8B active) | Open |
| 190 | Gemini 3.1 Flash-LiteGoogle | 11% | Not reported | Closed |
| 191 | Seed 1.6ByteDance | 11% | 230B (23B active) | Closed |
| 192 | baidu/ernie-4.5-vl-424b-a47b (xhigh)Baidu | 11% | 424B (47B active) | Unknown |
| 193 | GPT-5.4 nanoOpenAI | 10% | Not reported | Closed |
| 194 | Gemini 3 FlashGoogle | 10% | Not reported | Closed |
| 195 | DeepSeek V3.2DeepSeek | 10% | 685B (37B active) | Open |
| 196 | Claude 3 HaikuAnthropic | 10% | Not reported | Closed |
| 197 | Gemini 3 FlashGoogle | 10% | Not reported | Closed |
| 198 | nvidia/nemotron-3-nano-30b-a3b:free (xhigh)NVIDIA | 10% | 30B (3.5B active) | Unknown |
| 199 | Kimi K2Moonshot AI | 10% | 1,000B (32B active) | Closed |
| 200 | Grok 4.1 FastxAI | 10% | Not reported | Closed |
| 201 | MiniMax M2.5MiniMax | 9% | 229B (10B active) | Closed |
| 202 | Hy3Tencent | 8% | 295B (21B active) | Open |
| 203 | MiniMax M2.5MiniMax | 8% | 229B (10B active) | Closed |
| 204 | GLM-4.5Z.AI | 8% | 355B (32B active) | Closed |
| 205 | MiniMax M2.7MiniMax | 8% | Not reported | Open |
| 206 | DeepSeek-R1DeepSeek | 8% | 671B (37B active) | Open |
| 207 | o4-mini (high)OpenAI | 8% | Not reported | Closed |
| 208 | Seed 1.6ByteDance | 7% | 230B (23B active) | Closed |
| 209 | MiniMax M2.7MiniMax | 7% | Not reported | Open |
| 210 | DeepSeek-R1DeepSeek | 7% | 671B (37B active) | Open |
| 211 | prime-intellect/intellect-3 (low)Prime Intellect | 7% | 106B (12B active) | Unknown |
| 212 | mistralai/mistral-small-2603 (high)Mistral | 6% | 119B (6.5B active) | Unknown |
| 213 | qwen/qwen3-235b-a22b (none)Alibaba | 6% | 235B (22B active) | Unknown |
| 214 | GLM-4.5Z.AI | 6% | 355B (32B active) | Closed |
| 215 | GPT-OSS 120BOpenAI | 5% | 117B (5.1B active) | Open |
| 216 | nvidia/nemotron-nano-9b-v2:free (none)NVIDIA | 5% | 8.89B | Unknown |
| 217 | prime-intellect/intellect-3 (high)Prime Intellect | 5% | 106B (12B active) | Unknown |
| 218 | ai21/jamba-large-1.7AI21 Labs | 5% | 398B (94B active) | Unknown |
| 219 | o4-mini (high)OpenAI | 4% | Not reported | Closed |
| 220 | baidu/ernie-4.5-300b-a47bBaidu | 4% | 300B (47B active) | Unknown |
| 221 | deepseek/deepseek-chatDeepSeek | 4% | 685B (37B active) | Unknown |
| 222 | mistralai/mistral-small-2603 (none)Mistral | 4% | 119B (6.5B active) | Unknown |
| 223 | baidu/ernie-4.5-vl-424b-a47b (none)Baidu | 3% | 424B (47B active) | Unknown |
| 224 | qwen/qwen3-235b-a22b (xhigh)Alibaba | 3% | 235B (22B active) | Unknown |
| 225 | nvidia/nemotron-nano-9b-v2:free (xhigh)NVIDIA | 3% | 8.89B | Unknown |
| 226 | google/gemma-3-27b-itGoogle | 3% | 27B | Unknown |
| 227 | mistralai/mistral-large-2512Mistral | 2% | 675B (41B active) | Unknown |
| 228 | GPT-4o miniOpenAI | 2% | Not reported | Closed |
How BullshitBench v2 is shown here
BenchLM mirrors the published BullshitBench v2 leaderboard using the official snapshot generated on September 29, 2026 at 8:42 PM UTC. The public view reports per-model clear-pushback rates across 100 nonsense prompts, scored by a 3-judge panel.
BullshitBench is a useful reasoning sanity check, but BenchLM currently keeps it display only rather than weighted. The public leaderboard is highly variant-specific and exposes reasoning-effort settings directly, so BenchLM treats it as a mirrored external benchmark instead of a canonical ranking input.
Snapshot
The published BullshitBench v2 snapshot places Claude Opus 4.8 first at 95%. The third row is 1.0 points behind. The broader top-10 range is 15.0 points, so the table still separates the published systems.
228 models have been evaluated on BullshitBench v2. The benchmark falls in the Reasoning category. BullshitBench v2 is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About BullshitBench v2
Year
2025
Tasks
Nonsensical and flawed prompts across multiple domains
Format
Prompt challenge and refusal evaluation
Difficulty
Robustness and critical reasoning
BullshitBench evaluates a crucial real-world capability: knowing when NOT to answer. Models that score highly recognize flawed premises, impossible physics scenarios, and logical contradictions rather than hallucinating plausible-sounding responses. V2 includes harder and more diverse challenge categories.
Freshness and provenance
Version
BullshitBench v2 2025
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does BullshitBench v2 measure?
A benchmark that tests whether AI models challenge nonsensical, ill-posed, or logically flawed prompts instead of confidently generating incorrect answers. Measures the critical ability to push back on bad input.
Which model leads the published BullshitBench v2 snapshot?
Claude Opus 4.8 currently leads the published BullshitBench v2 snapshot with 95% clear pushback rate. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on BullshitBench v2?
The September 29, 2026 at 8:42 PM UTC snapshot contains 228 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,500+ readers.
One email each week. Unsubscribe anytime.