FrontierMemory: benchmarking continual learning through memory (FrontierMemory)
We show this table for reference; we do not rank on it.
Letta's benchmark of how good the memory is that an AI model writes from past agent work. The model studies a corpus of earlier agent transcripts and writes memory files; the score is the share of 15 memory-dependent coding and knowledge-work tasks that a task agent then fully passes with that memory, averaged over three runs.
Full-pass rate on FrontierMemory — FrontierMemory v1.1 (live leaderboard)
We mirror the published full-pass rate view for FrontierMemory. Claude Opus 5.5 leads the public snapshot at 73.3%, followed by Claude Fable 5.2 (73.3%) and DeepSeek V4.1 Flash 1017 (71.1%). We do not use these results to rank models overall.
Claude Opus 5.5
Anthropic
max reasoning
±0.0 SD over 3 runs · $3.42 per task
Claude Fable 5.2
Anthropic
max reasoning
±6.7 SD over 3 runs · $3.71 per task
DeepSeek V4.1 Flash 1017
DeepSeek
max reasoning
±3.8 SD over 3 runs · $3.86 per task
17 modelsAgenticCurrentDisplay onlyUpdated FrontierMemory v1.1 (live leaderboard)
Full-pass rate table (17 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Claude Opus 5.5Anthropicmax reasoning±0.0 SD over 3 runs · $3.42 per task | 73.3% | Not reported | Closed |
| 2 | Claude Fable 5.2Anthropicmax reasoning±6.7 SD over 3 runs · $3.71 per task | 73.3% | Not reported | Closed |
| 3 | DeepSeek V4.1 Flash 1017DeepSeekmax reasoning±3.8 SD over 3 runs · $3.86 per task | 71.1% | Not reported | Open |
| 4 | Gemini 4 CarbonGooglehigh reasoning±7.7 SD over 3 runs · $3.99 per task | 71.1% | Not reported | Closed |
| 5 | Kimi K3Moonshot AImax reasoning±3.8 SD over 3 runs · $4.16 per task | 71.1% | Not reported | Open |
| 6 | Claude Haiku 5.5Anthropicmax reasoning±3.8 SD over 3 runs · $3.98 per task | 68.9% | Not reported | Closed |
| 7 | GLM-5.3Z.AImax reasoning±3.8 SD over 3 runs · $4.08 per task | 68.9% | Not reported | Open |
| 8 | Gemini 4 ArgonGooglehigh reasoning±10.2 SD over 3 runs · $1.05 per task | 64.4% | Not reported | Closed |
| 9 | Gemini 4 HydrogenGooglexhigh reasoning±3.8 SD over 3 runs · $4.02 per task | 64.4% | Not reported | Closed |
| 10 | Inkling 2 SmallThinking Machines Labmax reasoning±6.7 SD over 3 runs · $4.18 per task | 60.0% | Not reported | Closed |
| 11 | Kimi K3.1 ProMoonshot AImax reasoning±3.8 SD over 3 runs · $3.61 per task | 55.6% | Not reported | Open |
| 12 | GPT-6.1 SolOpenAImax reasoning±3.8 SD over 3 runs · $4.00 per task | 51.1% | Not reported | Closed |
| 13 | Muse WonderMetamax reasoning±3.8 SD over 3 runs · $3.71 per task | 48.9% | Not reported | Closed |
| 14 | GPT-6.1 TerraOpenAImax reasoning±6.7 SD over 3 runs · $3.97 per task | 46.7% | Not reported | Closed |
| 15 | Muse EmberMetamax reasoning±3.8 SD over 3 runs · $3.60 per task | 42.2% | Not reported | Closed |
| 16 | Grok 4.7xAIxhigh reasoning±3.8 SD over 3 runs · $3.70 per task | 37.8% | Not reported | Closed |
| 17 | GPT-6 LunaOpenAImax reasoning±10.2 SD over 3 runs · $4.05 per task | 28.9% | Not reported | Closed |
FrontierMemory 1.0
BenchLM mirrors the table from Letta's FrontierMemory 1.0 post. It is excluded from BenchLM overall and category rankings.
| Source rank | Source model or configuration | Full-pass rate | Source setup | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | Claude Opus 5.5Anthropic | 73.3% | max reasoningOriginal result | Not reported | Closed |
| 2 | Kimi K3Moonshot AI | 71.1% | max reasoningOriginal result | Not reported | Open |
| 3 | GLM-5.3Z.AI | 68.9% | max reasoningOriginal result | Not reported | Open |
| 4 | Gemini 4 ArgonGoogle | 64.4% | high reasoningOriginal result | Not reported | Closed |
| 5 | Muse WonderMeta | 51.1% | xhigh reasoningOriginal result | Not reported | Closed |
| 6 | GPT-6.1 SolOpenAI | 51.1% | max reasoningOriginal result | Not reported | Closed |
| 7 | Grok 4.7xAI | 37.8% | xhigh reasoningOriginal result | Not reported | Closed |
How to read this leaderboard
Higher is better. Claude Opus 5.5 and Claude Fable 5.2 lead at 73.3% (33 of 45 task runs), ahead of DeepSeek V4.1 Flash, Gemini 4 Carbon and Kimi K3 at 71.1%. GPT-6 Luna is last at 28.9%. With 15 tasks and three runs, one task run moves a score by about 2.2 points, so models a few points apart are close; read the run spread (SD) beside each score.
Operator receipt: 17 sourced rows are currently displayable on this page; the leading published row is Claude Opus 5.5 at 73.3%.
Honest limit: Fifteen tasks is a small set, and the task agents are fixed (Claude Opus 5.5 and GPT-6 Sol), so a model's score reflects how useful its memory is to those agents, not to itself. Letta builds the default memory harness (Letta Dream) and task harness (Letta Code) and ranks first on its own harness board. The corpus changed on October 29, 2026, so v1.0 and v1.1 scores are not comparable. BenchLM shows FrontierMemory as display only; it does not enter overall or category rankings.
How FrontierMemory is shown here
We mirror the FrontierMemory model leaderboard from Letta. Each model reads a fixed corpus of past agent transcripts in Letta's Dream harness (up to 6 hours, no internet) and writes memory files. A task agent then works on 15 coding and knowledge-work tasks that need what was in those transcripts, with that memory loaded.
The score is the share of tasks fully passed, for the better of two task agents (Claude Opus 5.5 or GPT-6 Sol at max effort in Letta Code), averaged over three runs. So it rates the memory a model writes, not how the model does the tasks itself. Each row also gives the spread across runs and the average task cost. FrontierMemory is display only on BenchLM. The FrontierMemory 1.0 table from October 2026 sits below on this page.
Snapshot
The published FrontierMemory snapshot places Claude Opus 5.5 first at 73.3%. The third row is 2.2 points behind. The broader top-10 range is 13.3 points, so the table still separates the published systems.
17 models have been evaluated on FrontierMemory. The benchmark falls in the Agentic category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. FrontierMemory is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About FrontierMemory
Year
2026
Tasks
15 memory-dependent coding and knowledge-work tasks adapted from DeepSWE, SkillsBench, AutomationBench, APEX-Agents and EnterpriseOps-Gym
Format
Memory written from a corpus of past agent transcripts in Letta Dream (6-hour limit, no internet), then used by a task agent in Letta Code; three runs
Difficulty
Learning task conventions, user preferences and environment facts from past transcripts that the task prompt leaves out
Most memory benchmarks quiz an agent on facts it was told, such as a user's birthday. FrontierMemory instead asks whether memory makes an agent better at real work. Each memory model runs in Letta's Dream harness for up to 6 hours, with no internet, over a fixed corpus of agent transcripts (v1.1 since October 29, 2026; each transcript is marked as a success or failure, and only part of the corpus is relevant to the test tasks). It writes memory files: task conventions, user preferences and facts about the work environment. A task agent then works on 15 tasks adapted from DeepSWE, SkillsBench, AutomationBench, APEX-Agents and EnterpriseOps-Gym, changed so that solving them needs knowledge found only in the transcripts. Letta reviewed every task by hand and repaired broken verifiers. The task agent runs in Letta Code with Claude Opus 5.5 and with GPT-6 Sol at five effort levels; the leaderboard score is the better of the two at max effort, as a strict full-pass rate averaged over three independent runs. Letta also ranks memory harnesses (Letta Dream, Hermes Agent, Claude Managed Agents, Prime Agent, OpenAI Agents SDK) on the same tasks; BenchLM keeps that table in its snapshot but shows only the model board.
Freshness and provenance
Version
FrontierMemory 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does FrontierMemory measure?
How useful the memory is that an AI model writes from past agent work. The model reads a corpus of earlier agent transcripts and writes memory files; a task agent then tries 15 tasks that need what was in those transcripts. The score is the share of tasks fully passed, averaged over three runs.
Which model writes the best memory on FrontierMemory?
Claude Opus 5.5 and Claude Fable 5.2 tie at 73.3%, followed by DeepSeek V4.1 Flash, Gemini 4 Carbon and Kimi K3 at 71.1% (v1.1 corpus, read November 13, 2026).
How is FrontierMemory different from LongMemEval or LoCoMo?
Those benchmarks test recall: whether an agent can answer questions about facts from past conversations. FrontierMemory tests whether memory written from past work helps an agent solve new tasks, such as following a team's conventions or a user's preferences that the task prompt does not state.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.