# Controlled simulation validation v2 · official Pi runner

The simulator's usefulness is a hypothesis to test. This study does **not** start with an assumption that Jev/Laya and a target LLM are equivalent.

## Scope and shared calculation

The homepage supports template tools plus up to six added mock tools, each explicitly scheduled once before the answer. Users can edit low/default/high allowances for system framing, tool-result size, decision output and final-answer output. Those values are uncalibrated sensitivity assumptions, not confidence intervals. The exact fixture bill and the conditional target-LLM envelope are separate.

The browser and experiment both use `src/lib/estimate/budget-range.mjs`. Experiments reconstruct target-LLM payloads from a surrogate trajectory, then independently run the target LLM in the same world. Jev tokens are never repriced as LLM tokens.

Both policies run **the official `@earendil-works/pi-agent-core@0.87.1` Agent**, with `@earendil-works/pi-ai@0.87.1` pinned in the lockfile. The target LLM uses Pi’s OpenAI-compatible provider adapter with native function calling. A surrogate adapter converts each Jev/Laya Choice into a Pi tool call. Pi handles tool validation, execution, error messages and subsequent turns on both paths.

This is a constrained Pi execution, not the full Pi coding agent. Tools have a required fixture reference declared as a literal in their schema: the target emits it, and the surrogate binds it in code. Each turn allows one tool call; `finish`, `decline` and `ask_user` are explicit control tools. Final prose is not generated or billed. The experiment does not validate arbitrary-query decomposition, argument synthesis, free-text answer quality, final-answer token allowances, browser/web access, parallel tools, reasoning-model budgets or context compaction. The separate interactive sandbox remains at `/simulator/sandbox`.

## Run in GitHub Actions

Repository: `luhayes/jevcalc` → Actions → **Simulator validation** → Run workflow.

1. `mode=demo`, `suite=smoke`: no secrets or model calls. Confirms the controlled world, reports and artifact pipeline. It produces no measured cost-error result.
2. `mode=jev` or `laya`: choose `provider`, enter an exact supported target model ID, paste verified `prices_json`, and explicitly enable `confirm_paid`.
3. Start with smoke (4 calibration + 4 holdout tasks). Standard is 8 + 8. Each task runs both policies independently, up to 12 decisions each. Maximum 192 or 384 model requests respectively. There are no hidden provider retries. Longer or failed requests may still be billed.
4. Download `simulator-<mode>-<suite>-<run id>` from the run's Artifacts. Summary is also in the run page.

Live secrets are repository Actions secrets: `TYPESAFE_API_KEY` for Jev; target `GEMINI_API_KEY`, `DEEPSEEK_API_KEY` or `OPENAI_API_KEY`. The default frontend and demo pipeline do not receive these keys. No secrets belong in workflow inputs. Secrets on another repository are not automatically inherited.

The official Pi `openai-completions` adapter uses each provider’s OpenAI-compatible Chat Completions endpoint (including Gemini’s compatibility endpoint), streamed native tool calls, `temperature=0` and `max_tokens=256`. Select a model supporting those parameters and streamed usage. JSON-object response mode is no longer used. An incompatible model fails with a saved diagnostic; it does not silently fall back. Provider-specific hidden reasoning can affect billed output and invalidate a tight budget.

`prices.example.json` is a schema example, **not verified current pricing**. Replace both model IDs, date, source URL(s) and prices. Tariffs are USD per million tokens. Jev defaults to `jev-1.13.0`. Local Laya selects `typed-decisions`; input/output tariffs may be explicitly zero because it has no provider token bill, but CPU Actions minutes, downloads and hosting are outside the reported total.

Laya mode installs `laya[serve]==0.3.7` on the runner and starts its official localhost server. It uses the upstream typed-decisions weights. Package versions and model cache revisions are exported; weights can change upstream, so future bit-for-bit reproducibility is not promised. Laya's upstream context/option budgets may truncate state and must be checked when interpreting long tasks. A local Laya model failing to load is an environment failure, not model-quality evidence.

## Local commands

Requires Node 22.19 or newer.

```sh
npm ci
node --test experiments/simulator/test.mjs
SIM_MODE=demo SIM_SUITE=standard node experiments/simulator/run.mjs
# Optional safe extra tool JSON and token allowances:
SIM_MODE=demo SIM_TOOLS_FILE=experiments/simulator/tools.example.json node experiments/simulator/run.mjs
```

`SIM_ASSUMPTIONS_FILE` accepts the same four triplets as `DEFAULT_BUDGET`. `SIM_TOOLS_FILE` accepts at most six unique `custom_*` names, descriptions and static JSON results. These extra tools are optional candidates for both policies, not required ground truth. No code, URLs or arbitrary commands from the file are executed. Edit the versioned fixture environment and assertions when adding a new scored task. Example frontend tools are deliberately scheduled; benchmark tools are selected by the model, so the homepage behavior is not itself validated by a successful benchmark.

## Protocol

- English refund and document-research tasks; eligible, ineligible, missing, transient-error, empty-search and longer-document variants.
- Fresh world per policy, same fixture and action-specific failure schedule. Hidden outcomes do not appear in the initial state. Tool preconditions enforce mutations but do not tell the model which action to choose.
- Both policies share the task, candidate descriptions and rules for observing tool results. The target receives the full Pi transcript; the surrogate receives a structured projection of actual Pi tool-result messages. Their encodings differ, and independent decisions can produce different observations. Calls are sequential; policy order alternates per pair.
- On the surrogate path, the official target adapter serializes a hypothetical request and stops before HTTP transport. This captures the target’s tool schemas and history representation without making an extra target call. Token counts for these reconstructed payloads remain heuristic estimates, not provider measurements.
- Target usage is read from raw provider SSE usage, rather than SDK default zeros. Input includes cached tokens at the supplied standard tariff; output already includes reported reasoning tokens, which are not added again. Missing usage stays unknown.
- Business tool calls count attempted fixture operations. `piToolCallAttempts` also includes terminal control tools and attempts rejected by Pi. Both are saved so they are not confused. Pi events and messages are snapshotted from the actual runner.
- Calibration and holdout IDs, fixture contents and payload sizes differ. They share task families; this is in-domain evaluation, not unseen-domain generalization.
- Baseline prediction uses only surrogate state/path plus frozen assumptions and tariffs. No target trajectory is an input to prediction.
- Calibrate only a median target/prediction cost ratio. Save the correction before collecting any holdout run. Holdout never updates it. Each new protocol revision needs fresh holdout cases if previous holdout results informed tuning.
- Save raw and corrected error. Report all-task completion and also the both-completed cost subset. A cheap incomplete run is not a saving.
- Unknown usage remains unknown. Report cost coverage and known subtotals, not a fabricated full bill.

## Artifacts and publication

`manifest.json`: configuration, tariffs, Pi version, lockfile hash, code/version and dataset hashes.
`calibration.json`: frozen correction, sample size and timestamp.
`pairs.jsonl`: per-pair traces with Pi events/messages, actual or reconstructed native target payloads, tool results, usage and failures.
`report.json` / `report.md`: completion, mean turns/tools, turn/tool MAE, raw/corrected median absolute percentage cost error, signed bias and envelope coverage.
`cost-comparison.csv`: chart-ready predictions and measured target costs.
`article-draft.md`: evidence-linked draft scaffold. Do not automatically publish it.

Only synthetic fixtures are logged; do not replace them with private production data in public artifacts. Provider error bodies and credentials are not logged.

The public article is a **method note** until a live run is reviewed. Add measured charts and conclusions only from a linked immutable run. Publish negative results too. Small samples cannot establish equivalence, calibrated 90% coverage, or accuracy for arbitrary prompts.

Version 2 changes the runner and target request protocol. Do not reuse version 1 calibration factors. The dataset size is unchanged: offline tests establish runner behavior, not prediction accuracy.