LLM prices fell up to 80%. A model-driven test run is cheaper now, and still a sample

Max Woolf's timeline of price cuts is on the Hacker News front page this morning, and for anyone who runs tests with a model in the loop, it changes one line of the budget. Re-driving a 50-flow suite with a frontier model on every commit used to be an expensive idea. It is now a cheap one. Our view: that removes the weaker objection to model-driven test runs and leaves the stronger one standing. A run driven by a model is a sample, and a cheaper sample is still a sample.

What changed

Woolf's post tracks OpenAI and Anthropic list prices from June to October. The headline numbers:

• GPT-5.6 Luna was cut by 80% in July.
• GPT-6.1 Sol now costs $2 per million input tokens and $0.10 per million cached reads, against $5 and $0.50 for GPT-5.6 Sol in June.
• Claude Opus 5.5 lists at $4 input and $0.20 cached read, against $5 and $0.50 for Opus 5 in July. The pricing page confirms it.

His point about agents matters most here. In agentic coding, he writes, cache reads are "the vast majority (96% - 99%) of the total token expenditure", so cutting the cache-read price cuts the real bill far more than the input price suggests. A model re-driving a browser flow is exactly that kind of loop: the same page context, read again on every step.

What a re-drive costs now

We priced one realistic setup. The per-flow figure comes from our own benchmark, where re-driving one known flow with a model took about 30,249 tokens (scorecard). The suite is 50 flows, run 40 times a working day. We priced input tokens only, at two bounds: all uncached, and all cached reads. Output tokens are left out, so treat these as floors.

At cached rates, a model re-drive of the whole suite on every commit now costs about $6 to $12 a day. If cost was your reason for not letting an agent re-check every flow on every push, the reason is gone.

The thread saw where this leads. vikramkr argued that unattended work is where usage goes: "Agents in the background, long running tasks, unattended tasks, all that stuff is what is even capable of driving significant usage" (comment). A test suite re-driven on every commit is the most unattended job there is.

What did not change

Price is per token. Variance is per run, and nothing in this timeline touches it.

A model-driven check reads the page, decides what to click, and decides whether the outcome looks right. Each of those is a sample. Thinking Machines Lab measured how far that goes even at temperature 0: sampling 1,000 completions of one prompt from one model, they got "80 unique completions", and only got identical outputs after rewriting the inference kernels (post). Most hosted APIs do not run in that mode.

In a test suite, that variance has a name: flake. Here is the arithmetic, as an illustration rather than a measurement. If each model re-drive disagreed with itself on 1% of runs, a 50-flow suite would show at least one spurious result on about 40% of runs (1 minus 0.99 to the 50th). A suite that is red two runs in five gets retried until it is green, and then nobody reads it.

So the price cut moves the bottleneck. When a re-drive cost $7.56, teams ran it rarely and read the result. At $0.15, they will run it on every push, and the cost they feel will be the noise.

What to do with the cheaper tokens

Spend them where a sample is what you want. Exploring a changed flow, proposing test cases, and reading a diff for risk are all jobs where a different answer each run is a feature: you want coverage, not agreement.

Keep sampling out of the gate. The check that decides whether a pull request merges should give the same answer for the same code, every run. That means recording what the flow should do once, with a model if you like, and replaying it as assertions on requests and state. Reticle works this way: in its benchmark, eight replays of one flow gave one verdict, and a four-flow suite re-verifies in about 480 tokens with no model in the loop.

The useful question after this round of cuts is not "can we afford to have the model check everything?" You can. It is "which of our checks are allowed to change their mind?"