Company Agent Bench
We show this table for reference; we do not rank on it.
Concept's benchmark of how well an AI agent understands the company it works in. Agents answer 205 questions drawn from one month of Concept's own Slack history: who wrote a message, who was tagged and who sent each reply. The score is overall accuracy.
Overall accuracy on Company Agent Bench — Launch post, Nov 12, 2026
We mirror the published overall accuracy view for Company Agent Bench. Claude Fable 5.2 leads the public snapshot at 82.2%, followed by GPT-6 Astra (80.8%) and Claude Opus 5.5 (79.2%). We do not use these results to rank models overall.
Claude Fable 5.2
Anthropic
high reasoning
API model in a Slack harness · low 78.9% · medium 80.0% · $1.345 per question at high
GPT-6 Astra
OpenAI
high reasoning
API model in a Slack harness · low 76.7% · medium 81.1% · $0.557 per question at high
Claude Opus 5.5
Anthropic
high reasoning
API model in a Slack harness · low 54.5% · medium 72.5% · $0.441 per question at high
12 modelsAgenticCurrentDisplay onlyUpdated Launch post, Nov 12, 2026
Overall accuracy table (12 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Claude Fable 5.2Anthropichigh reasoningAPI model in a Slack harness · low 78.9% · medium 80.0% · $1.345 per question at high | 82.2% | Not reported | Closed |
| 2 | GPT-6 AstraOpenAIhigh reasoningAPI model in a Slack harness · low 76.7% · medium 81.1% · $0.557 per question at high | 80.8% | Not reported | Closed |
| 3 | Claude Opus 5.5Anthropichigh reasoningAPI model in a Slack harness · low 54.5% · medium 72.5% · $0.441 per question at high | 79.2% | Not reported | Closed |
| 4 | GPT-6.1 SolOpenAIhigh reasoningAPI model in a Slack harness · low 73.9% · medium 78.8% · $0.108 per question at high | 79.1% | Not reported | Closed |
| 5 | DotsOpenAIAgent product · default settings · Authorship 72.0% · Mentions 74.0% · Reply matching 81.8% | 75.9% | Not reported | Unknown |
| 6 | GPT-6.1 TerraOpenAIhigh reasoningAPI model in a Slack harness · low 70.3% · medium 76.8% · $0.036 per question at high | 74.3% | Not reported | Closed |
| 7 | Claude Tag (Fable 5.1)AnthropicAgent product · default settings · runs Claude Fable 5.1 · Authorship 69.0% · Mentions 74.0% · Reply matching 76.4% | 73.1% | Not reported | Unknown |
| 8 | Grok BotxAIAgent product · default settings · Authorship 67.0% · Mentions 68.0% · Reply matching 83.0% | 72.7% | Not reported | Unknown |
| 9 | Claude Sonnet 5.5Anthropichigh reasoningAPI model in a Slack harness · low 35.6% · medium 52.7% · $0.147 per question at high | 68.7% | Not reported | Closed |
| 10 | Claude Tag (Opus 5.5)AnthropicAgent product · default settings · runs Claude Opus 5.5 · Authorship 64.0% · Mentions 66.0% · Reply matching 73.3% | 67.8% | Not reported | Unknown |
| 11 | DevinCognitionAgent product · default settings · Authorship 60.0% · Mentions 68.0% · Reply matching 70.3% | 66.1% | Not reported | Unknown |
| 12 | Claude Tag (Sonnet 5.5)AnthropicAgent product · default settings · runs Claude Sonnet 5.5 · Authorship 58.0% · Mentions 60.0% · Reply matching 70.9% | 63.0% | Not reported | Unknown |
How to read this leaderboard
Higher is better. Among the products, Dots leads at 75.9%, ahead of Grok Bot (72.7%), Claude Tag with Opus 5.5 (67.8%) and Devin (66.1%). Dots leads Authorship (72.0%) and Mentions (74.0%); Grok Bot leads Reply matching (83.0%). The strongest API models in Concept's harness scored higher than every product: Fable 5.2 at 82.2% and GPT-6 Astra at 80.8% at high effort. Effort matters: Opus 5.5 scores 54.5% at low effort and 79.2% at high, and GPT-6 Astra peaks at medium (81.1%).
Operator receipt: 12 sourced rows are currently displayable on this page; the leading published row is Claude Fable 5.2 at 82.2%.
Honest limit: The test set is one small company's Slack over one month, about 10 people, so results may not carry over to a larger team or another chat tool. Concept publishes no dataset, code or live leaderboard, and BenchLM transcribed the scores from the post's charts. Each product was scored on one Slack replay at default settings, and the post does not name the model behind Dots, Grok Bot or Devin. API rows reflect Concept's own harness. BenchLM shows Company Agent Bench as display only; it does not enter overall or category rankings.
How Company Agent Bench is shown here
Concept built Company Agent Bench from one month of its own Slack history, with about 10 teammates, and replayed it in sandbox Slack workspaces in date order. The 205 questions ask who wrote an anonymized message (Authorship, 100 questions), who was tagged (Mentions, 50) and who sent each reply in a thread (Reply matching, 55 panels). An agent sees only what was posted before the message that gives the answer. The score is overall accuracy, with the three parts weighted equally.
The board has two kinds of rows. Agent products (Dots, Grok Bot, Claude Tag and Devin) were connected to Slack at their default settings. API models ran in Concept's shared Slack harness with native Slack search and read tools; these rows are their high-effort runs, and each row's label gives low and medium effort. Claude Tag appears once for each model it ran: Opus 5.5 (the post's default view), Sonnet 5.5 and Fable 5.1.
The test set is one small company's Slack, so a larger team or another chat tool could give different results. Concept publishes no dataset or live leaderboard, so BenchLM transcribed the charts in the post. Company Agent Bench is display only on BenchLM.
Snapshot
The published Company Agent Bench snapshot places Claude Fable 5.2 first at 82.2%. The third row is 3.0 points behind. The broader top-10 range is 14.4 points, so the table still separates the published systems.
12 models have been evaluated on Company Agent Bench. The benchmark falls in the Agentic category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. Company Agent Bench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About Company Agent Bench
Year
2026
Tasks
205 questions from one month of Concept's Slack: 100 Authorship, 50 Mentions and 55 Reply matching panels
Format
Replay of the Slack history in sandbox workspaces; agent products at default settings, API models in a shared Slack harness
Difficulty
Knowing who owns what, who works with whom and how each teammate thinks, from Slack alone
Concept recreated its channels, people and conversations in sandbox Slack workspaces and replayed one month of history in date order. Each agent answers with its own search and memory and sees only what was posted before the message that gives the answer. Authorship (100 questions) asks who wrote an anonymized message, Mentions (50) asks who was tagged, and Reply matching (55 panels) asks which teammate sent each reply in a thread, scored as the share of replies matched and averaged across panels. For Authorship and Reply matching, Concept standardized spelling and grammar to remove typing habits as a clue. The overall score weights the three parts equally. Four agent products ran at default settings: Dots, Grok Bot, Claude Tag and Devin. Six API models (GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, Claude Fable 5.2, Opus 5.5 and Sonnet 5.5) ran in a shared Slack harness with native Slack search and read tools at low, medium and high reasoning effort; this page shows their high-effort runs, as Concept's headline chart does. Claude Tag ran with three models; Opus 5.5 is its default view in the post.
Freshness and provenance
Version
Launch post (November 12, 2026)
Refresh cadence
On source changes
Staleness state
Current
Question availability
Questions private; drawn from Concept's own Slack, with nine examples in the post
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does Company Agent Bench measure?
Whether an AI agent connected to a company's Slack understands the people in it: who wrote a message, whom a teammate would tag and who sent each reply in a thread. Concept built the 205 questions from one month of its own Slack history.
Which agent product scores highest on Company Agent Bench?
Dots, at 75.9% overall. Grok Bot scores 72.7%, Claude Tag with Opus 5.5 67.8% and Devin 66.1%. All four ran at default settings.
Do API models beat the agent products?
In Concept's own Slack harness, the strongest did. Fable 5.2 at high effort scored 82.2% and GPT-6 Astra 80.8%, above Dots at 75.9%. At low effort, Opus 5.5 (54.5%) and Sonnet 5.5 (35.6%) scored below all four products.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.