Can a small decision model
predict agent costs?

The question worth testing

A detailed bill is useful only if its assumed execution is plausible. Similar classification scores do not establish similar tool choices, stopping behavior or task completion. We test whether a surrogate policy produces a useful cost estimate under an explicit environment and target-model configuration.

The current simulator supports editable tools and low/default/high token allowances. These are scenario assumptions. The default price and token values are illustrative; the displayed range is not a statistical confidence interval.

One world, two independent runs

Refund tasks include eligible, expired, missing-order and transient-error cases. Research tasks include complete collections, empty search, read failures and longer documents. Models see facts only after retrieving them. Tool preconditions prevent invalid mutations but do not select the correct next action for the model.

Reconstruct requests before estimating costs

We do not multiply Jev usage by an LLM tariff. Each surrogate turn is serialized by the official Pi provider adapter into a target-LLM request containing instructions, tool schemas and observed history, without sending that request. Token counts for these payloads are estimates; they are not provider measurements. The shared estimator adds explicit allowances for framing, tool-result length and output. Live target costs use provider-returned usage at the supplied tariffs.

Known fixture content is a floor. Increasing tool-output allowances increases later context as well as the current observation. The browser and validation harness share the conditional-envelope calculation. Missing usage stays unknown, including potentially billed failed requests.

Freeze calibration, then open the holdout

The smoke suite contains four calibration and four held-out tasks; the standard suite contains eight of each. They are disjoint synthetic instances within the same task families. This is a small in-domain diagnostic, not a generalization benchmark for arbitrary prompts.

1. Collect paired calibration runs and their raw predictions.
2. Fit one transparent correction: the median ratio of measured target cost to predicted cost.
3. Write the correction and dataset hash before collecting held-out runs.
4. Report raw and corrected held-out errors without refitting.

If held-out results influence a later change, that revision needs fresh held-out tasks. Reusing a test split after tuning would weaken the evidence.

What the report must show

The workflow exports per-turn traces, exact target payloads, a frozen calibration file, JSON and Markdown reports, chart-ready CSV and an unpublished article draft. A conclusion should link the immutable workflow run and state the sample size, model IDs, tariff date and limitations.

What this first experiment cannot establish

Both policies now run the official Pi Agent engine, pinned to version 0.87.1. The target uses native function calling through Pi’s OpenAI-compatible adapter; Jev/Laya choices are converted into Pi tool calls. Pi validates and executes tools, records errors and advances the loop on both paths. This aligns the execution engine, not the policies’ behavior.

We constrain each turn to one call, bind arguments to fixture references and use explicit terminal control tools. This does not test the full Pi coding agent or arbitrary argument generation. Final prose is not generated; answer quality and answer-generation costs remain outside the measured scope.

It also does not validate arbitrary-query plan generation, live web tools, reasoning models, parallel tool execution or context compaction. Custom benchmark tools return static JSON and cannot execute code or call external services. Local Laya compute and Actions minutes are excluded from provider-token fees.

Model confidence is not used as a measured retry probability. Monte Carlo sampling can be added after credible behavior distributions exist; extra samples alone would not repair incorrect assumptions.