# Offline framework replay

This directory runs the same 360 saved attempts and 12 authored negative controls through Promptfoo and DeepEval. It reads the files already included in `support-extraction-v1`; it does not call a model, regenerate outputs or download another corpus.

The replay is deliberately narrow. It shows how two frameworks express this kit's schema and exact-field checks. It is not a general framework benchmark, quality score or winner comparison.

## Download the complete kit

There is no separate archive. The public manifest is the file allowlist and gives the byte count and SHA-256 for every download. These commands fetch the manifest and acquisition helper over HTTPS, verify the helper before executing it, then download every allowlisted file into a new directory while preserving its manifest path:

```bash
KIT_BASE='https://benchlm.ai/downloads/evidence-kits/support-extraction-v1'
BOOTSTRAP_DIR="$(mktemp -d)"
curl --fail --location --proto '=https' --proto-redir '=https' --tlsv1.2 "$KIT_BASE/manifest.json" -o "$BOOTSTRAP_DIR/manifest.json"
curl --fail --location --proto '=https' --proto-redir '=https' --tlsv1.2 "$KIT_BASE/frameworks/download-kit.mjs" -o "$BOOTSTRAP_DIR/download-kit.mjs"
node -e 'const fs=require("node:fs"),c=require("node:crypto");const [m,s]=process.argv.slice(1),j=JSON.parse(fs.readFileSync(m)),e=j.files.find(x=>x.path==="frameworks/download-kit.mjs"),b=fs.readFileSync(s),h=c.createHash("sha256").update(b).digest("hex");if(!e||b.length!==e.bytes||h!==e.sha256)throw Error("download helper does not match manifest")' "$BOOTSTRAP_DIR/manifest.json" "$BOOTSTRAP_DIR/download-kit.mjs"
node "$BOOTSTRAP_DIR/download-kit.mjs" --base-url "$KIT_BASE" --output "$PWD/support-extraction-v1"
cd support-extraction-v1/frameworks
```

`download-kit.mjs` rejects unsafe or duplicate paths, unexpected sizes and hashes, oversized manifests/files, an existing output path, requests that exceed 15 seconds and HTTPS-to-HTTP redirects. Response bodies are read through hard byte caps even when `Content-Length` is absent. HTTP is accepted only for numeric loopback addresses so the acquisition test can use a local static server. It downloads into a staging directory and renames it only after every file verifies. The saved `manifest.json` is the manifest used for verification.

These SHA-256 checks establish byte consistency and integrity against the manifest obtained from the same HTTPS origin. They are not an independent authenticity proof if that origin or connection is compromised.

## Tested versions

| Component | Tested version | License |
|---|---:|---|
| Node.js | 24.19.0 | — |
| Promptfoo | 0.123.1 | MIT |
| Python | 3.12.14 | — |
| DeepEval | 4.2.3 | Apache-2.0 |

DeepEval 4.2.3 is not claimed to work on every Python version in its declared package range. This replay was exercised on Python 3.12.14.

## Install in this directory

Installation requires network access to npm and PyPI. It does not modify the website application's dependencies.

```bash
npm install --prefix .node --no-save --ignore-scripts --no-audit --no-fund promptfoo@0.123.1
python3.12 -m venv .venv
.venv/bin/pip install --disable-pip-version-check -r requirements.txt
```

After installation, the replay itself requires no network, API key, model provider or LLM judge. The runner disables framework telemetry, update checks, sharing, remote generation and cache. A host-level firewall or network sandbox is the enforcement boundary if you require proof of blocked egress; environment flags alone are configuration, not a firewall.

## Frozen replay, not an update loop

Run `support-extraction-v1` as shipped. Its fixture, schema, controls, saved attempts, oracle, source hashes and expected outcome contract are one frozen evidence unit. A source-hash failure means the local inputs differ; it is not an invitation to replace the expected hash.

For another fixture, schema or workload, fork a separately named kit outside `support-extraction-v1`. Review its expected answers and rules independently, define a new provenance contract, and record a new expected outcome contract before drawing conclusions. Do not edit this fixture or update hashes/outcomes until failures disappear: those failures are evidence, and changing the oracle to make a replay green invalidates the comparison.

## Run both frameworks twice

From this `frameworks` directory:

```bash
node run-offline.mjs
```

With no `--work-dir`, the runner atomically creates a unique directory under the system temporary directory and prints its path. To select a location, pass a leaf path that does not exist; its parent must already exist:

```bash
node run-offline.mjs --work-dir ./replay-work-1
```

The runner refuses an existing path before it launches any replay command. It never clears or reuses a work directory and never deletes caller-owned data. A failed replay leaves its newly created directory available for inspection; choose another new path for the next attempt.

To choose existing isolated installations explicitly:

```bash
node run-offline.mjs --work-dir ./replay-work-2 \
  --promptfoo /absolute/path/to/promptfoo \
  --python /absolute/path/to/python
```

The runner:

1. verifies the installed Promptfoo and DeepEval versions and the downloaded fixture, schema, controls, oracle and two saved-attempt files against `expected-results.json`;
2. builds a local 372-row corpus in the work directory;
3. runs Promptfoo twice and normalizes each raw result;
4. runs DeepEval twice;
5. requires exact unique record sets, independently recomputes every outcome hash, compares every row with the existing oracle and checks the curated expected counts.

The printed work directory contains the derived corpus and framework receipts. They are local work products, not additional downloads and not evidence to republish.

## Exit codes

Promptfoo exits `100` because this corpus deliberately contains negative controls, saved schema/task failures and 12 saved provider errors. The runner may continue after that specific exit only so the fail-closed normalizer and comparator can inspect the result. **Exit `100` does not mean parity passed.** Only a final comparator exit of `0` verifies that the exact record sets, counts, repeats and oracle outcomes match.

The runner exits non-zero for an unexpected framework exit, a wrong source hash, a missing/duplicate/extra result, a stale reported hash, an oracle difference, a repeat difference or an expected-count difference.

## Expected bounded result

Each of the four normalized replays should report:

| Rows | Completed | Transport failures | Schema pass | Schema fail | Task pass | Task fail | Task not run |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 372 | 360 | 12 | 350 | 10 | 326 | 24 | 22 |

The normalized outcome SHA-256 is `365b87d593bd8554bc609df6af65d6691f83fbb229d3bbf3df8732f8ad00171b`.

The denominator matters: 372 rows means 360 saved attempts, including 12 recorded transport failures, plus 12 authored controls. The combined `completed = 360` is 348 completed saved attempts plus 12 controls.

## What is native and what is adapted

| Check | Promptfoo | DeepEval | Boundary |
|---|---|---|---|
| Load saved output | custom JavaScript provider | adapter creates `LLMTestCase` | custom, local adapter |
| JSON Schema | native `is-json` | native `JsonCorrectnessMetric` | native scoring |
| Exact four-field task check | JavaScript assertion runner | custom `BaseMetric` | deterministic custom logic |
| Saved transport failure | provider `error` | adapter failed row | differently expressed |
| Schema gate, repeat comparison and oracle hash | normalizer/comparator | runner/comparator | post-processing |

The source schema declares JSON Schema 2020-12 and has a stable `$id`. Promptfoo 0.123.1's exercised AJV path did not load that meta-schema and registered repeated IDs globally, so `promptfooconfig.cjs` removes only the `$schema` and `$id` annotations. It preserves every validation keyword.

DeepEval's JSON score is deterministic, but optional invalid-schema reasons can initialize a model path. This replay sets `include_reason=False` and supplies a fail-closed model whose generation methods only raise. Both DeepEval metrics must return a literal Python boolean from `is_successful()`; unexpected null, numeric, string or container values stop normalization instead of being coerced to a negative result. No LLM judge is used.

## Files

| File | Purpose |
|---|---|
| `build-corpus.mjs` | verifies source hashes and derives the local replay corpus |
| `download-kit.mjs` | downloads every manifest entry into a new directory and verifies size and SHA-256 |
| `promptfoo-provider.cjs`, `promptfooconfig.cjs` | Promptfoo saved-output provider and assertions |
| `normalize-promptfoo.mjs` | fail-closed raw-result normalization |
| `deepeval_runner.py` | native JSON metric plus deterministic exact-task metric |
| `compare-results.mjs` | exact-set, hash, repeat, framework and oracle comparison |
| `run-offline.mjs` | two-replay orchestration and exit handling |
| `expected-results.json` | curated fixture-scoped hashes and result counts |
| `requirements.txt` | exact DeepEval top-level dependency pin |

Promptfoo and DeepEval retain their respective MIT and Apache-2.0 licenses. This directory contains BenchLM-authored adapters and no vendored framework code.