# The evaluator must expose the failure

> Compare two LLM evaluation frameworks on one 372-row saved-output replay, then choose by stack, failure inspection, and maintenance.

- Published: 2026-09-22
- Last reviewed: 2026-09-22
- Data as of: 2026-09-22
- Article slug: llm-evaluation-frameworks
- Author: [Glevd](https://x.com/glevd)
- Reading time: 13 minutes
- Topics: llm-evaluation, promptfoo, deepeval, structured-output, testing
- Data policy: Dated analysis; static benchmark claims are retained.
- Benchmark catalog updated: September 25, 2026
- Canonical URL: https://benchlm.ai/llm-evaluation-frameworks

**Start with Promptfoo when an evaluation belongs in configuration and JavaScript. Start with DeepEval when it belongs in Python.**

The better choice is the one whose failed case your team can still explain six months later.

On September 22, 2026, we replayed one 372-row saved-output corpus twice through Promptfoo 0.123.1 and twice through DeepEval 4.2.3. All four normalized runs matched the reference assertions row for row and produced the same outcome hash. No model generated a new answer. No LLM judged one.

Scope stops there. The replay compares two implementation surfaces and cannot name an overall winner.

## A failure record beats a feature list

Feature lists are the wrong first screen for an evaluation tool. A team does not debug “evaluation support.” It debugs one case: the output that arrived, the check that failed, and the reason the check gave.

Our minimum useful record has five parts:

1. a stable case identifier.
2. whether a response completed or failed in transport.
3. whether the output passed the schema.
4. whether a schema-valid output passed the task rule.
5. a deterministic reason for the failed check.

Failure reasons change the decision. A red total can mean malformed JSON, the wrong category, or no response at all. Each failure has a different owner and a different fix. Roll them into one score and the framework has made the report tidy by making the work harder.

We used a synthetic support-ticket extraction fixture because the expected answer can be written down. Each output has four fields: `ticket_id`, `category`, `urgency`, and `requested_action`. Schema validation checks the container. Task validation compares each value with a reviewed answer. The [structured-output test note](/blog/posts/test-structured-output) explains why both passes are necessary. The [custom benchmark guide](/blog/posts/building-custom-llm-benchmark) owns the wider job of designing the fixture and deciding whether a model should change.

Framework selection starts after those rules exist. If the rule is still “the answer looks good,” neither tool can rescue it.

## Four replays have the same 372-row outcome set

Our corpus contains 360 saved model attempts from two saved runs and 12 authored negative controls. Twelve saved attempts are recorded transport failures, so “360 completed” in the combined receipt means 348 completed saved attempts plus all 12 controls. It does not mean 360 successful model responses.

Both frameworks reproduced these normalized counts on both replays:

| Outcome | Count | Denominator and meaning |
| --- | --- | --- |
| Rows | 372 | 360 saved attempts plus 12 authored controls |
| Completed | 360 | 348 saved responses plus 12 completed controls |
| Transport failures | 12 | Attempted saved rows with no model output |
| Schema pass | 350 | Completed rows whose output matched the schema |
| Schema fail | 10 | Completed rows rejected by the schema check |
| Task pass | 326 | Schema-valid rows matching all four expected fields |
| Task fail | 24 | Schema-valid rows with at least one wrong field |
| Task not run | 22 | 12 transport failures plus 10 schema failures |

Every normalized run produced SHA-256 `365b87d593bd8554bc609df6af65d6691f83fbb229d3bbf3df8732f8ad00171b`. The comparator recomputed that hash from the four outcome fields for every case, checked the exact 372 identifiers, and rejected missing, duplicate, extra, or altered rows. Nine corruption and happy-path tests passed around those boundaries.

Oracle parity is not parity between every Promptfoo and DeepEval feature. These two adapters reproduced this rule over this corpus. The result does not say that either tool is more accurate, faster, safer, or easier to operate.

Installation needed network access. Once Promptfoo and DeepEval were installed, the saved-output replay did not. We ran an additional full replay of each framework in the default network-restricted task sandbox. Both produced the same normalized hash. The public [framework replay README](/downloads/evidence-kits/support-extraction-v1/frameworks/README.md) keeps the offline commands beside the fixture.

## The same rule still needs three implementation layers

“Both frameworks ran the same checks” leaves out the part that matters.

Each implementation has a native schema check, a custom task check, and some BenchLM-authored adapter or post-processing code. The boundary is different in each tool.

| Layer | Promptfoo 0.123.1 | DeepEval 4.2.3 |
| --- | --- | --- |
| Load saved output | Custom JavaScript provider | Python adapter creates an<br><br>`LLMTestCase` |
| Validate schema | Native<br><br>`is-json`<br><br>assertion | Native<br><br>`JsonCorrectnessMetric` |
| Compare four task fields | JavaScript assertion runner executes our deterministic function | Our<br><br>`ExactTaskMetric`<br><br>subclasses<br><br>`BaseMetric` |
| Stop task reporting after schema failure | Our normalizer sets the task result to null | Our runner does not invoke<br><br>`ExactTaskMetric` |
| Preserve a saved transport failure | Provider returns an error | Adapter records an incomplete row before metrics |
| Check repeats and oracle parity | Our comparator | Our comparator |

Promptfoo's configuration really does execute two assertions:

```javascript
assert: [
  { type: 'is-json', value: promptfooSchema },
  { type: 'javascript', value: taskExact },
]
```

But Promptfoo does not natively turn the second result into `null` when the first fails. Both assertions execute. Our normalizer discards the task component after a schema failure so the public tri-state rule remains intact. Calling that automatic metric dependency would assign our post-processing to the framework.

Promptfoo's exercised AJV path also rejected the fixture's JSON Schema 2020-12 declaration and reused `$id`. Our adapter removed only `$schema` and `$id`. Required keys, types, patterns, enums, `additionalProperties: false`, and `minLength` stayed unchanged. Removing two annotations preserved the contract.

DeepEval's runner makes the gate explicit:

```python
schema_metric = JsonCorrectnessMetric(
    expected_schema=SupportExtraction,
    model=OfflineGuardModel(),
    include_reason=False,
    async_mode=False,
    strict_mode=True,
)

if schema_metric.is_successful():
    task_metric = ExactTaskMetric()
    task_metric.measure(case)
```

DeepEval documents JSON correctness as deterministic, while its optional explanation for an invalid schema can call a model. We set `include_reason=False` and supplied an `OfflineGuardModel` whose generation methods raise an error. An unexpected judge path would fail the replay instead of quietly making a call.

Shared logic can produce shared outcomes. It cannot prove native feature equivalence.

## One failed case beats the total

Aggregate parity matters because it catches drift. The case record matters because someone has to fix the drift.

`short:B:case-11:r1:a1` completed and produced valid JSON. It failed the task because the response supplied the wrong category. The normalized comparison agrees on this row's identifier, completion state, schema result, and task result. Separately, all four framework receipts preserve the same task reason. That reason is shown below for inspection but is not part of the normalized outcome hash.

```json
{
  "record_id": "short:B:case-11:r1:a1",
  "completed": true,
  "schema_pass": true,
  "task_pass": false,
  "task_reason": "Exact task mismatch: category"
}
```

Reviewers now know where to look. The record also avoids a common reporting mistake: this output did not fail JSON and should not be grouped with ten schema failures. Category equality failed.

Transport is different again. Promptfoo expressed a saved gateway refusal as a provider error. The standalone DeepEval adapter had no provider transport layer, so it recorded the same saved fact as an incomplete row before any metric ran. The normalized outcome agrees, but the native event does not. A reader choosing between tools needs both facts.

We did not launch either framework's GUI or reproduce a dashboard for the article. The machine-readable record is the evidence. A polished screenshot would make the comparison easier to glance at and harder to rerun.

## Promptfoo fits the config and JavaScript path

Promptfoo is the direct starting point when a repository already maintains JavaScript, configuration files, and CLI checks. The framework owns the `is-json` assertion and the execution of the JavaScript task assertion. The adapter owns saved-output loading, and the normalizer owns the tri-state report.

Keeping both assertions together lets a reviewer see the schema and task rules in one configuration. It also has a cost. Our parity receipt depends on code outside the native CLI summary, because the summary reports 326 passes, 34 assertion failures, and 12 provider errors. Expected failures produce those totals. Promptfoo exits with status 100. Our comparator's zero exit is the signal that the replay matched the oracle.

Choose this path if your CI already treats an expected-failure corpus as an artifact rather than assuming any non-zero tool exit means the job broke. Keep the raw report and normalized receipt separate. Raw files contain run-specific IDs and timestamps, so byte-for-byte equality is the wrong repeat test. The normalized case set and recomputed hash are the stable comparison.

Promptfoo loses this recommendation when the team does not own JavaScript or when the adapter and normalization step would become orphaned glue. A compact config is not lower maintenance if nobody reviews the code it calls.

Official [assertion documentation](https://www.promptfoo.dev/docs/configuration/expected-outputs/) and [JavaScript provider documentation](https://www.promptfoo.dev/docs/providers/custom-api/) cover the native surfaces. Our replay README covers only the adapter we exercised.

## DeepEval fits the Python path

DeepEval is the direct starting point when evaluation already lives beside Python tests and Pydantic models. `JsonCorrectnessMetric` owns schema scoring. `ExactTaskMetric` is our custom deterministic metric, and the runner decides whether it is called.

Runner control is visible, which we prefer to pretending every check ran. When JSON correctness fails, `task_pass` remains null and the receipt says the task was gated off. When transport failed in the saved run, the adapter records the row before constructing any metric. Neither behavior should be attributed to a universal DeepEval scheduler.

There are two operational catches in the exercised version. First, `include_reason=True` can involve a model when the schema is invalid, so an offline deterministic run has to disable those explanations and fail closed around the model interface. Second, the DeepEval 4.2.3 package metadata declares Python 3.9 support, but its import failed on our Python 3.9 host before corpus data loaded. Python 3.12.14 worked. That is one host observation, not a claim that every Python 3.9 installation fails.

We ran the metrics as a standalone local pipeline rather than through `deepeval test run`. That kept the JSON receipt local and avoided hosted reporting, but it also means our comparator supplies the report and parity checks. Do not read its zero exit as a native DeepEval test-suite result.

DeepEval loses this recommendation when Python is foreign to the application repository or when the team expects model-generated explanations without accepting their network, cost, and second-model dependency. Official [JSON correctness](https://deepeval.com/docs/metrics-json-correctness) and [custom metrics](https://deepeval.com/docs/metrics-custom) pages document the two metric surfaces. They do not validate our adapter choices for another workload.

## The replay can't name an overall winner

Our replay does not cover RAG relevance, agent trajectories, red-team cases, conversational turns, subjective writing quality, model-judge reliability, production traces, setup time, runtime speed, or total cost. Feature pages can document support for some of those jobs. They cannot turn this structured-output replay into evidence about them.

It also does not compare the products' hosted experiences. Telemetry and sharing were disabled, no application account was connected, and no GUI was opened. The [observability guide](/blog/posts/best-llm-observability-tools) owns the separate decision about traces, production feedback, and experiment systems.

Even within this fixture, the two integrations do different work around transport and reporting. That is not a flaw in the comparison. It is the comparison. Framework code, custom code, and post-processing should be named separately so a team knows which part it will maintain.

If the next evaluation needs an LLM judge, design a new test. Measure that judge against authored controls and human review before its score decides anything. The deterministic result here offers no shortcut.

## Fork the fixture for the useful next step

Download scope is deliberately smaller than article scope. The bundle contains the reviewed adapter and configuration surface, plus instructions for replaying the already public synthetic fixture. It does not export private raw framework receipts or the internal comparison corpus.

Start with the [framework replay README](/downloads/evidence-kits/support-extraction-v1/frameworks/README.md):

1. Install the pinned Promptfoo and DeepEval dependencies. Installation needs network access.
2. Run the authored negative controls and confirm each broken output fails the intended check.
3. Replay the saved outputs locally. Once dependencies exist, the replay needs no provider key or network call.
4. Open a schema failure, a task failure, and a transport failure. Confirm each has a different state.
5. Leave the frozen fixture, hashes, and replay contract unchanged. To use your own data, fork a separate workload evaluation instead of editing the replay in place.
6. Give the fork approved, scrubbed examples. Independently review its expected answers and rules, then establish a new provenance contract and expected outcome contract before running a model.
7. Version the fork's fixture, schema, task check, adapter, framework version, and receipt together.

If you later generate new outputs, pin the API identifier in the [model ID directory](/model-ids), record the route and retry policy, and use the current [pricing table](/llm-pricing) only for a dated cost calculation. Those facts were outside this replay because no provider call occurred.

A framework earns its place when the next failed case is cheaper to explain than to ignore.

## Frequently asked questions

### How should I choose an LLM evaluation framework?

Choose the framework whose test surface and failure record your team can maintain. Promptfoo fits a configuration, CLI and JavaScript workflow. DeepEval fits a Python evaluation workflow. Before deciding, run an intentionally broken case and confirm the saved result shows the case, failed check and reason.

### Can I use deterministic assertions instead of an LLM judge?

Yes, when the expected answer can be written exactly. Schema validity, identifiers, enum labels and quoted fields can use deterministic checks. Open-ended summaries or style judgments need human review or a calibrated model grader. This comparison used deterministic checks only and made no judge or provider call.

### What should be versioned with an LLM evaluation?

Version the fixture, expected answers, schema, custom checks, framework version, adapter, normalization code and run receipt. Keep transport failures and skipped checks visible. A pass total without the underlying case IDs and reasons cannot show whether a later code change altered the rule or only the report.

### Should a JavaScript team choose Promptfoo or DeepEval?

Promptfoo is the more direct starting point when configuration files, a CLI and JavaScript assertions already fit the repository. DeepEval remains viable, but it adds a Python environment and runner. The reverse applies to a Python team. This is a maintenance recommendation, not a framework-quality ranking.

### What does this Promptfoo and DeepEval comparison not test?

It does not test generation quality, RAG metrics, agent traces, red teaming, model judges, latency, cost, onboarding time or production monitoring. It replays one synthetic structured-output corpus through two deterministic implementations. The result establishes reproducibility for that workflow and nothing broader about either framework.