Twelve Prompt Fixes, Including One Regression

The most carefully rewritten prompt in this test made its output worse. We ran 12 before-and-after AI prompt examples three times per variant on openai/gpt-5.6-luna: across the 10 noncoding cases whose work could be checked from text, rough prompts fully passed 20 of 30 runs and revised prompts passed 28 of 30 — but the launch announcement regressed from 3/3 to 1/3. Four cases improved, five tied, one got worse.

The useful outcome was not a more polished prompt. It was more cited, complete, correctly shaped work with fewer invented fields — and evidence that added detail is a cost until a failed check justifies it. Two coding examples remained excluded because text cannot prove that repository tests passed.

We froze the work before the wording

Every case began with a public synthetic input, one rough prompt, and acceptance conditions chosen before the revised output existed. The model route, omitted temperature, 2,500-token limit, and three-run count stayed fixed. OpenRouter returned Azure for all 72 calls in this artifact.

The revised prompts followed one deliberately plain rule: name the task, list the requirements, refuse unsupported guesses, and return only the requested work product. That repair is not sophisticated. Its value comes from the frozen check waiting on the other side.

Table 1 Download CSV

Case	Rough prompt	Repair added	Rough full	Revised full	Result
Launch announcement	“Make it compelling and concise”	Facts, prohibited claims, shape, call to action	3/3	1/3	Regression
Scoped date bug	“Fix the date bug”	Behavior, allowed files, commands	Excluded	Excluded	Repository required
JSON export	“Add JSON export”	Existing CSV contract, visible rows, ISO dates	Excluded	Excluded	Repository required
Invoice JSON	“Extract important details”	Exact keys, no extras, null rule	3/3	3/3	Tie
Support routing	“Tell me who should handle it”	Label allowlist, quoted evidence, confidence	3/3	3/3	Tie
Weekly conversion	“Explain what happened”	Rate formula, comparison, unavailable drivers	3/3	3/3	Tie
Latency anomalies	“Find anything unusual”	Volume floor, p50/p95 split, no cause claim	3/3	3/3	Tie
Provider policies	“Compare and recommend”	Four criteria, section citations, unknown rule	0/3	3/3	Improved
Research evidence	“Summarize these studies”	One row per study, six columns, no pooling	0/3	3/3	Improved
Executive summary	“Make this concise”	Five required facts, status boundary, 160 words	2/3	3/3	Improved
French release note	“Translate and make natural”	Preserve placeholders, product name, Markdown	3/3	3/3	Tie
Meeting actions	“Turn notes into action items”	Owner, action, due date, source evidence	0/3	3/3	Improved

The frozen fixtures, public inputs, and 72-run artifact preserve every prompt, output, check, latency, token count, and provider cost.

Six repairs that changed the work

The provider-policy request failed for an ordinary reason: “compare” did not say which differences controlled the decision, and “recommend” did not require evidence. All three rough outputs discussed the policies, but none satisfied the four-criterion citation contract.

The revised prompt named retention, training use, regional processing, and enterprise controls. It required a source ID and section heading for each claim, preserved unknown values, and tied the recommendation to a workload assumption. All three revised outputs then met the contract.

The research-summary case exposed a different failure. A prose summary made three incompatible studies sound more comparable than they were. The repair required one row per study with population, task, comparison, outcome, sample size, and limitation. It also prohibited a pooled effect. The result changed from 0/3 to 3/3 without asking the model to sound more expert.

Meeting notes needed an uncertainty rule. One action had an owner and date, another had neither, and a third had an owner but no date. The rough outputs filled the table but missed source evidence and the explicit absence of fields. The repaired prompt required Unassigned and No date set instead of a guess. Three partial results became three full results.

The executive summary moved from 2/3 to 3/3 after the prompt protected five facts that “concise” could otherwise erase: customer impact, duration, root-cause status, mitigation, and the next decision. This was the smallest observed gain, and it matters. One accepted result is still one less summary that needs reconstruction.

Invoice extraction and support routing did not improve because the source input already supplied much of the contract. It listed the invoice keys and missing-value behavior; the support ticket input included allowed labels, evidence, and confidence fields. Both variants passed 3/3. Repeating those rules in the instruction added tokens, not value.

The same pattern held for the weekly conversion, latency, and localization cases. Complete source material can do more work than elaborate prompt prose.

The detailed launch prompt got worse

The launch announcement is the case a polished before-and-after list would be tempted to drop. We kept it.

The rough request was:

Write a launch announcement for our API cost dashboard. Make it compelling and concise.

The source input separately supplied four approved facts, three prohibited claim categories, and the required call to action. All three rough outputs passed the adjudicated checks.

The revised prompt repeated the prohibited categories and added a generic rule to mark unsupported claims unknown. Two outputs obeyed literally by writing that customer adoption, savings percentages, and performance improvements were unknown. The draft did not invent a claim, but it repeated language the publication contract said to keep out.

More explicit was worse here.

The repair for the next run would be smaller: “Do not mention prohibited categories, including to deny or disclaim them.” That instruction follows the observed failure. Adding more general caution would only make the prompt longer.

Coding needs a repository, not a prose receipt

The two coding cases are present in the raw run, but their text completions do not count toward the 20/30 or 28/30 totals. Their acceptance plans required changed files plus test, lint, and build results. A model saying those commands passed is not evidence that they did.

We later ran four coding tasks inside a frozen dependency-free repository. Both prompt variants passed every command in all 12 runs. The difference appeared in review scope, not automated correctness: the rough feature and refactor requests widened the task in an unblinded post-run review. The full method belongs in Coding Changes That Survive Review.

That separation protects the customer outcome. A coding prompt is not successful because it generated plausible code. The patch must survive the repository.

Use the smallest repair that survives

A useful prompt revision closes one observed decision:

1. Freeze a representative source input.
2. Define what accepted work looks like.
3. Run the rough prompt more than once.
4. Find the requirement that failed.
5. Add the smallest instruction or source field that protects it.
6. Rerun the same case under the same settings.
7. Keep ties and regressions in the record.

The AI Prompt Optimizer shows this kind of repair as a diff; the AI Prompt Generator covers the case where no prompt exists yet. Neither replaces the frozen check that decides whether the rework actually shrank.

Model route also changes the result and cost. Compare current API pricing, then test the route you will actually use. This run cost $0.073731 for 72 completions, but one cheap synthetic batch is not a production cost forecast.

The next useful edit is the one attached to a failed requirement. Stop when the same work passes without it.