Simulator validation methodology

The simulator's usefulness is a hypothesis to test. This study does not start with an assumption that Jev/Laya and a target LLM are equivalent.

Scope and shared calculation

The homepage supports template tools plus up to six added mock tools, each explicitly scheduled once before the answer. Users can edit low/default/high allowances for system framing, tool-result size, decision output and final-answer output. Those values are uncalibrated sensitivity assumptions, not confidence intervals. The exact fixture bill and the conditional target-LLM envelope are separate.

The browser and experiment both use src/lib/estimate/budget-range.mjs. Experiments reconstruct target-LLM payloads from a surrogate trajectory, then independently run the target LLM in the same world. Jev tokens are never repriced as LLM tokens.

Both policies run the official `@earendil-works/pi-agent-core@0.87.1` Agent, with @earendil-works/pi-ai@0.87.1 pinned in the lockfile. The target LLM uses Pi’s OpenAI-compatible provider adapter with native function calling. A surrogate adapter converts each Jev/Laya Choice into a Pi tool call. Pi handles tool validation, execution, error messages and subsequent turns on both paths.

This is a constrained Pi execution, not the full Pi coding agent. Tools have a required fixture reference declared as a single-value enum with its exact value in the schema description: the target emits it, and the surrogate binds it in code. Each turn allows one tool call; finish, decline and ask_user are explicit control tools. Final prose is not generated or billed. The experiment does not validate arbitrary-query decomposition, argument synthesis, free-text answer quality, final-answer token allowances, browser/web access, parallel tools, reasoning-model budgets or context compaction. The former interactive sandbox has been retired; /simulator provides the offline assumption-based cost simulator.

Run in GitHub Actions

Repository: luhayes/jevcalc → Actions → Simulator validation → Run workflow.

1. mode=demo, suite=smoke: no secrets or model calls. Confirms the controlled world, reports and artifact pipeline. It produces no measured cost-error result.

2. mode=jev or laya: choose provider, enter an exact supported target model ID, paste verified prices_json, and explicitly enable confirm_paid.

3. Start with smoke (4 calibration + 4 holdout tasks). Standard is the original 8 + 8; fresh is a new set of 8 + 8 instances. Each task runs both policies independently, up to 12 decisions each. Maximum 192 or 384 decisions respectively; including retries, the request caps are 576 (smoke) and 1152 (standard/fresh). HTTP 429/502/503/504 allow at most two additional attempts per decision. SDK automatic retries remain disabled. Longer or failed requests may still be billed.

4. Download simulator-<mode>-<suite>-<run id> from the run's Artifacts. Summary is also in the run page.

Live secrets are repository Actions secrets: TYPESAFE_API_KEY for Jev; target GEMINI_API_KEY, DEEPSEEK_API_KEY or OPENAI_API_KEY. The default frontend and demo pipeline do not receive these keys. No secrets belong in workflow inputs. Secrets on another repository are not automatically inherited.

The official Pi openai-completions adapter uses each provider’s OpenAI-compatible Chat Completions endpoint (including Gemini’s compatibility endpoint), streamed native tool calls, temperature=0 and max_tokens=256. Select a model supporting those parameters and streamed usage. JSON-object response mode is no longer used. An incompatible model fails with a saved diagnostic; it does not silently fall back. Provider-specific hidden reasoning can affect billed output and invalidate a tight budget.

prices.example.json is a schema example, not verified current pricing. Replace both model IDs, date, source URL(s) and prices. Tariffs are USD per million tokens. Jev defaults to jev-1.13.0. Local Laya selects typed-decisions; input/output tariffs may be explicitly zero because it has no provider token bill, but CPU Actions minutes, downloads and hosting are outside the reported total.

Laya mode installs laya[serve]==0.3.7 on the runner and starts its official localhost server. It uses the upstream typed-decisions weights. Package versions and model cache revisions are exported; weights can change upstream, so future bit-for-bit reproducibility is not promised. Laya's upstream context/option budgets may truncate state and must be checked when interpreting long tasks. A local Laya model failing to load is an environment failure, not model-quality evidence.

Local commands

Requires Node 22.19 or newer.

npm ci
node --test experiments/simulator/test.mjs
SIM_MODE=demo SIM_SUITE=standard node experiments/simulator/run.mjs
# Optional safe extra tool JSON and token allowances:
SIM_MODE=demo SIM_TOOLS_FILE=experiments/simulator/tools.example.json node experiments/simulator/run.mjs

SIM_ASSUMPTIONS_FILE accepts the same four triplets as DEFAULT_BUDGET. SIM_TOOLS_FILE accepts at most six unique custom_* names, descriptions and static JSON results. These extra tools are optional candidates for both policies, not required ground truth. No code, URLs or arbitrary commands from the file are executed. Edit the versioned fixture environment and assertions when adding a new scored task. Example frontend tools are deliberately scheduled; benchmark tools are selected by the model, so the homepage behavior is not itself validated by a successful benchmark.

Protocol

Artifacts and publication

manifest.json: configuration, tariffs, Pi version, lockfile hash, code/version and dataset hashes.

calibration.json: frozen correction, sample size and timestamp.

pairs.jsonl: per-pair traces with Pi events/messages, actual or reconstructed native target payloads, tool results, usage and failures.

report.json / report.md: completion, mean turns/tools, turn/tool MAE, raw/corrected median absolute percentage cost error, signed bias and envelope coverage.

cost-comparison.csv: chart-ready predictions and measured target costs.

article-draft.md: evidence-linked draft scaffold. Do not automatically publish it.

Only synthetic fixtures are logged; do not replace them with private production data in public artifacts. Provider error bodies and credentials are not logged.

The public article is a method note until a live run is reviewed. Add measured charts and conclusions only from a linked immutable run. Publish negative results too. Small samples cannot establish equivalence, calibrated 90% coverage, or accuracy for arbitrary prompts.

Version 2 changes the runner and target request protocol. Do not reuse version 1 calibration factors. The dataset size is unchanged: offline tests establish runner behavior, not prediction accuracy.

Gemini compatibility correction (pi-v2.1)

Gemini tool-call extra_content.google.thought_signature is retained from raw streaming chunks and replayed unchanged on the matching historical call. This bridges metadata omitted by the pinned generic Pi adapter; the runner remains official Pi. Signatures are never invented, and opaque signature values are excluded from heuristic textual token estimates. Surrogate paths do not have Gemini signatures and remain counterfactual request estimates, not executable Gemini conversations.

The fixed fixture argument now uses a single-value enum with an explicit description, with the same strict local validation. HTTP status and bounded diagnostic categories appear in console logs, failures.json and the job summary; raw provider errors and credentials remain excluded. Prior results with protocol errors do not establish target-model capability. Recollect calibration after this request-schema correction. Model loops reaching the turn cap remain task failures; the change does not force a successful policy path.

Separate estimator error from trajectory differences

Every new run exports diagnostics.json, diagnostics.md and diagnostics.csv, embeds diagnostics in report.json, and adds tables to the Actions summary. The three cohorts are all held-out tasks, both policies completed, and at least one policy failed. Every cohort reports its task and comparable-pair counts, raw/calibrated median APE, signed mean error and scenario-envelope coverage.

Two retrospective content-only estimates use the same shared budget calculation:

For each task, the raw signed USD identity is:

surrogate estimate - actual = (target-path estimate - actual) + (surrogate estimate - target-path estimate)

The first component is the estimator residual; the second is the trajectory/content difference under the estimator. This is an arithmetic diagnostic, not a causal attribution. Percentage-error medians are not additive. A second decomposition in JSON applies the same target-path calibration factor to both estimates. Accuracy tables also retain separately calibrated estimates, each fitted only on calibration cases. Do not subtract their errors to assign causality.

Both calibration factors are saved before holdout collection in new runs. Missing content or usage stays unknown. Percentage errors require positive actual cost; zero-cost examples may still enter dollar differences. Failed-task contribution means its share of summed absolute raw dollar residuals, not contribution to the median percentage error.

Reanalyze an existing artifact without model calls

Unzip the original Actions artifact, then run from the repository root:

node experiments/simulator/analyze.mjs --input /path/to/extracted-artifact --output /path/to/new-diagnostics --source-run https://github.com/luhayes/jevcalc/actions/runs/RUN_ID

The command reads only manifest.json and pairs.jsonl, preserves source files, requires no API key and never imports or executes the provider runner. It writes three diagnostic files into a separate directory. Provenance includes source hashes, collection/analysis revisions, tariff and budget settings. Offline factors are fitted on calibration rows after collection, and explicitly labeled retrospective; they were not frozen before the original holdout run.

An example reanalysis of run 35886868984 is committed under docs/simulator-validation/runs/35886868984/. Its held-out results have already been examined; it is a diagnostic record, not a new untouched validation set. Changes informed by these results need fresh holdout tasks. standard includes the existing smoke cases and is not a wholly new independent dataset.

Content accounting and explicit loop assumptions (content-heuristic-v2)

Input is now counted from system instructions, tool definitions and message content, rather than the complete transport JSON. Model names, stream settings, token limits, call IDs, provider usage and opaque signatures are excluded. Tool arguments and returned content remain included. The heuristic uses one token per CJK character and one per four other characters, rounded per component; LLM message framing is explicitly assumed to be 3 tokens/message plus 3/request. This is not a provider-specific tokenizer or a universal token-count standard.

Each request counts its actual retained history once. Context increases when another result is appended; the sum of input over sequential turns includes the repeated history. There is no fixed exponential-growth multiplier and no free summarization. Default system extras and minimum output/result length targets are now zero: zero preserves observed fixture content instead of padding each historical tool result to 400 tokens. Positive targets add only the difference on later requests. High assumptions remain editable sensitivity allowances, not statistical confidence bounds. Low and default may coincide when no extra allowance is configured.

The homepage and /simulator expose Extra loops (0–20) and Failed attempts before retry succeeds (0–5). They repeat the template's first context tool. Each failure/repeat creates an actual mock turn, model request and tool charge, and carries its observation into subsequent context. This is an explicit scenario, not a predicted failure rate. Per-turn fallback selections reset when these structural assumptions change. No API retry, real payment or external tool execution is performed.

Fresh validation instances

Select suite=fresh in Simulator validation. It contains 8 calibration and 8 holdout tasks with new IDs, order ages, document counts, lengths and prompt wording. It reuses the same two task families and world rules, so it is an in-domain validation set. As introduced, no live outcomes from these instances were used to tune the estimator. The original smoke/standard sets remain available for regression diagnostics. Fresh is at most 384 decisions / 1152 request attempts across both policies; paid authorization and repository secrets are required for live mode.

To compare the current content estimator/default allowances on an old artifact, explicitly override the original budget:

node experiments/simulator/analyze.mjs --input /path/to/extracted-artifact --output /path/to/content-v2-diagnostics --assumptions experiments/simulator/assumptions.content-v2.json

The output records both source and override budgets. The committed example runs/35886868984-content-v2/ uses this override. It is a retrospective regression diagnostic on already-inspected tasks, not a fresh accuracy claim; original reports remain unchanged. New calibration must be collected for the changed estimator/assumptions. Do not reuse old scale factors or treat improved point error as calibrated interval coverage.

Bounded provider retries

Both Jev/Laya HTTP requests and target-LLM HTTP requests use the same explicit policy: retry only 429, 502, 503 or 504, with at most three attempts total per decision. Backoff starts at 1 and 2 seconds plus up to 250 ms jitter; Retry-After is respected up to 30 seconds. Longer Retry-After values stop retries rather than sending early. Task cancellation and timeouts interrupt backoff. Authentication/validation errors, network exceptions, partial SSE failures and invalid model outputs are not retried. The SDK retry count remains zero.

A retry resends the same request within the same Pi turn, before the failed response reaches the agent. It cannot repeat a completed tool execution. Provider retries are distinct from simulated tool failures, which remain ordinary agent turns. The 12-decision limit remains; the manifest also records the expanded request-attempt cap. No whole-task restart or automatic paid workflow rerun occurs.

request-attempts.json records task, partition, policy, turn, attempt index, HTTP status, sanitized diagnostic, elapsed milliseconds, scheduled backoff, observed usage and cost. pairs.jsonl embeds the same records in turn.requestAttempts. turn.usage is final-response usage; turn.costUsd covers all attempts. Task/collection known-cost subtotals include successful attempts in otherwise unknown-cost turns. Missing usage is never assumed to be free, including an HTTP failure followed by success. Therefore a task can complete while its total bill is unknown and it is excluded from comparable cost pairs. Cost-coverage denominators remain explicit: task costCoverage is the fraction of fully costed turns; attemptCostCoverage covers HTTP attempts. Turn elapsed time includes all attempts and waits.

The report exposes retries and recovered decisions instead of erasing transient failures. Preserve each run's original artifact, including unsuccessful runs; a new paid validation is a separate run. This handling changes transport resilience, not the completion oracle or the cost estimator. Scenario envelopes remain editable assumptions, not statistical prediction intervals.

Content matrix and editable execution assumptions

The homepage now uses explicit pre-run steps and numeric budgets; see /research/execution-assumptions-method.md. Existing content-based reports do not validate that draft generator.

Select suite=matrix for 32 calibration + 32 held-out research tasks across English/Chinese prose, JSON/code, two content lengths, two document counts and clean/retry cases. Maximum 1536 decisions / 4608 attempts including retries. Source bodies and IDs are split-disjoint; the world rules and generation patterns are shared. This is synthetic in-domain stress evidence, not an arbitrary-query benchmark. Reports include strata, worst error and total-budget error. Live matrix results are pending; demo runs do not establish accuracy.

Download Markdown source