Start with a reviewable plan
“Read ten contracts and prepare a report” leaves important cost drivers unspecified. How long is each contract? Does the agent retain all prior text? How many calls are needed, and how long should the answer be?
The simulator now exposes those choices before playback. Quantities extracted from the task show their source phrase. Default budgets remain labeled as defaults. Add or reorder tools, repeat a stage, set result and output sizes, or choose a sliding context window. A window discards information; it is not a free summarization step.
Keep an inspectable ledger
Each turn lists the query, instruction budget, tool definitions, retained observations and accepted responses. Unknown text is represented as a numeric budget rather than invented document content. Costs are the sum of these input/output budgets at the declared rates, plus tool fees. A manually selected fallback bills both model requests while executing the tool once.
Jev applies only where the user declares a bounded decision. This setting is an architectural hypothesis, not a claim of equal task quality. Final generation remains on the LLM. Query tokenization is still a character heuristic; declared token budgets, framing and illustrative rates can all differ from a real deployment.
Why a small median error is not enough
Our completed controlled study had eight held-out tasks, full completion and full usage coverage. Raw median cost error was 4.2%, or 1.4% after calibration. Yet one long-input task was overestimated by 26.6% before calibration. A global correction made that case worse.
That study used observed mock content in a pinned Pi tool world. It tested a narrower question than predicting the cost of an unseen task before execution. It does not validate the new draft generator, unknown real tool results or final prose generation.
Source run and artifacts (repository access required)Test different questions separately
| Layer | Question | Status |
|---|---|---|
| Accounting | Do the declared calls, history and rates reconcile? | Offline software checks |
| Content / trajectory | How do estimates compare with independent Pi runs? | Small study complete; 64-task content matrix prepared |
| Prospective planning | Does a frozen pre-run draft resemble actual execution? | Broad evaluation still required |
The next content matrix varies English, Chinese, JSON and code, document length, document count and tool failures. It has 32 calibration and 32 held-out tasks. Live results are pending. These remain synthetic research tasks; increasing their number does not turn them into evidence about every agent domain.
Reports show error by content type and length, worst-case error and aggregate budget error alongside medians. Future prospective studies must freeze assumptions before executing the target agent, keep user-reviewed and automatically drafted plans separate, and hold out task families.
Inspect the assumptions. Then test them.
Use the calculator for scenario exploration now. Treat accuracy claims as conditional on the tested tools, models and workloads. No paid inference is triggered by opening this page or the editor.