How We Keep Pricing Data Current

The current pricing pipeline does not scrape provider pages and publish whatever it finds. A person checks an official first-party pricing page or announcement, maps the exact SKU into the curated registry, records the source-grounded note, and runs validation. Automation begins downstream: history, live tables, freshness checks, and generated pages rebuild from the accepted registry.

That boundary is slower than unattended scraping and much harder to embarrass.

This article contains partner links, marked before you reach them. Partner status never changes our data, validation rules, or recommendation. See the affiliate disclosure.

Browse AI is a partner and a no-code extraction and monitoring product we cover below. We do not currently route production provider prices through it. The web-scraping roundup compares its documented fit with Firecrawl, Apify, and code-first options.

The registry is the source of truth

Provider pages disagree more often than a clean pricing table suggests. A launch post may quote a family range, the API page may split cache hits from cache misses, and a model name in a dashboard may not be a separately priced SKU. Extracting every number perfectly would not resolve those identity questions.

Our pricing policy therefore starts with the exact row:

Table 1 Download CSV

Question	Required answer
Which model or API SKU is this?	Exact public identifier, not a family guess
Who published the price?	First-party provider source
What is the unit?	Currency, tokens or characters, and quantity
Which tier applies?	Standard, batch, cached, regional, or another named tier
Is a numeric row actually published?	If not, keep it unresolved and explain why
When was it checked?	Source note and registry update date

The refusal is part of the data. When a provider publishes only a range, a non-USD schedule, or a family statement that does not map to the exact SKU, the numeric fields stay unresolved. We do not borrow a sibling's price to fill the hole.

That rule matters more than collection speed. A fast wrong join is still wrong.

What is automated after review

Once a row survives source and identity review, the mechanical work should not depend on someone remembering five commands.

official source → human-reviewed registry change → validation → history snapshot → live tables and pages

The accepted pricing registry feeds the public LLM pricing table, provider pricing hubs, cost tools, and build-time article tables. History generation records a dated change for tracked price-series constituents. Freshness checks fail loudly when the registry or a living pricing article falls outside its allowed window. Generated blog data and public pages then rebuild from the same accepted row.

This design contains a useful asymmetry. A new value needs human approval once. Every downstream copy of that value should update mechanically.

It also has an honest limit: the pipeline cannot publish a change nobody has reviewed into the registry. Automated downstream work is not automated source discovery. We prefer a visible freshness failure to a silent crawler confidently attaching a cache price to the wrong model.

Extraction and monitoring are supporting jobs

Extraction answers, "What did this page contain when we captured it?" Monitoring answers, "What changed since the previous capture?" Neither answers, "Should this become the canonical price?"

A no-code monitor can still earn its keep. Browse AI's current official documentation describes recorded robots, scheduled monitors, workflows, webhooks, spreadsheets, and a REST API. A reasonable use would be to flag a changed pricing card for review. A bad use would be to let that diff overwrite the registry and deploy without checking the exact SKU and tier.

We have not wired that use into the production pipeline, so this is an architecture option rather than a case study. That correction matters because the earlier version of this article said we ran Browse AI against provider pricing pages. We do not.

If recorded robots and scheduled monitoring fit that pre-review job, evaluate Browse AI against one non-critical source first. Partner link; the capture still needs the same registry and review gate.

If source discovery becomes the bottleneck, the safe integration point is before review:

monitor alert → raw capture → human verifies official source → registry change

The monitor may shorten detection time. It does not inherit publishing authority.

Validation catches different mistakes than a scraper

HTML extraction failures are obvious when a field disappears. Semantic failures are worse: the parser returns a valid number attached to the wrong thing.

The checks need layers:

Table 2 Download CSV

Check	Example failure
Required fields and types	Output price arrives as an empty string
Unit normalization	Per-thousand price treated as per-million
Exact identity	"Pro" family rate attached to a preview SKU
Range and delta	A decimal shift creates a 100× price change
Duplicate keys	Two aliases overwrite the same canonical row
Source and timestamp	A number cannot be traced to a current provider page
Public freshness	A page still claims a current month after the registry aged out

Rows that fail should be quarantined, not coerced. Store the raw evidence before normalization so a reviewer can see whether the page changed, the parser changed, or the assumption was wrong.

The pipeline also separates price from benchmark ranking. A cheaper model does not receive a better capability score, and a benchmark update does not rewrite price. The price-performance page joins those datasets at presentation time without confusing their provenance.

The same discipline applies to RAG data

A RAG corpus has more rows and usually weaker source boundaries. That makes the registry pattern more important, not less.

Start with a source manifest: URL or feed, owner, license or permission basis, expected cadence, content type, and exclusion rules. Then preserve the retrieval time and source identity on every document. Deduplicate before chunking, keep document versions, and test whether the crawler covered the site rather than trusting a successful job status.

The acquisition tool can return markdown or typed JSON. It cannot prove that the corpus is complete, licensed for the intended use, or useful for answering the questions users ask. Choosing a RAG model comes after those checks. A stronger generator does not repair a missing source document.

For a small documentation corpus, a managed crawler may be sensible. For structured first-party APIs, skip scraping and use the API. For a handful of high-stakes prices, manual source review may still be the cheaper system once the cost of one wrong public number enters the calculation.

A small pipeline worth copying

The useful starter project is not "scrape every provider." It is one source with a refusal path.

1. Register one official page, the exact fields expected, and its authority.
2. Capture the raw page or API response with a timestamp.
3. Parse into a staging row without overwriting production.
4. Reject missing fields, unknown units, and a deliberately inserted outlier.
5. Require a person to approve the staged change.
6. Generate one downstream table from the accepted row.
7. Fail the build when the published freshness claim exceeds the evidence.

The deliberate outlier is important. A validation layer that has never rejected anything is either perfect or decorative. Bet on decorative.

For a no-code version of steps two and three, test one Browse AI robot and keep its output in staging. Partner link; do not grant the robot publishing authority.