# From a task to an explainable cost scenario Version: execution-assumptions/v1. Updated: 2026-09-24. ## What the product does The homepage accepts a task and creates an editable local-rules draft. This is not an LLM planner and it does not infer an arbitrary agent's true execution path. Recognized English/Chinese quantities are annotated with their source phrase. Steps, tool choices, missing sizes and outputs remain visible assumptions. Unrecognized tasks receive a generic answer stage that users must extend. The user can add/remove/reorder stages, specify repeated calls, mark bounded decisions, and choose full or sliding-window history. Explicit token values are budgets, not tokenizer measurements. Word counts and document counts are not silently interpreted as token counts. The draft and all field provenance can be exported/imported as versioned JSON. Pricing and policy are separate settings. Input is processed locally; no prompt is sent to a model or external tool when drafting or simulating. ## Model allocation Selecting a Jev strategy defaults unreviewed tool-selection stages to Jev under a visible bounded-choice, code-supplied-arguments assumption. It does not establish model suitability. Explicit user assignments are preserved, and importing a plan retains its saved allocation; model-only work remains on the LLM unless explicitly assigned, and final deliverables always use the LLM. The allocation editor and canvas show planned Jev/LLM calls even after playback finishes. No eligible assignment means an explicitly labeled LLM-only scenario. ## Accounting contract For each sequential stage repetition, the ledger adds: - Query tokens from the disclosed character heuristic. - Declared system instructions and combined tool/criteria budget. - Retained prior tool-result tokens and accepted reply tokens. - Explicit assumed message framing (LLM only). - The stage's declared output budget. Jev retains the same declared history content. There is no automatic context reduction or quality claim. Fallback adds a second model request (and a 32-token explanation assumption) but only one accepted reply and one tool effect. A stage's returned content enters subsequent requests, not the request that produced it. A sliding window discards old turns; it does not generate an unbilled summary. Requests over the editable input budget are flagged, never silently truncated. Interactive limits: 16 stages / 256 turns, with explicit errors beyond those bounds. Cost is sum(input tokens * input tariff + output tokens * output tariff) / 1e6 plus tool calls * tool tariff. Rates are editable, illustrative USD assumptions. Hidden reasoning, cache discounts, provider retries, parallelism and hosting are outside this ledger. A plan may fit the budget and still fail the real task. ## Layout and entry points The homepage opens Agent simulator by default; Single call remains available through the mode switch and `?view=single`. On desktop, the execution canvas and a scrollable turn inspector share the top row. A full-width cost module below places billing details beside the cost curve. Narrow screens stack these sections without an inner inspector scroll area. ## Cost curve The cost panel plots estimated cumulative costs or per-turn costs from the same ledger as the bill. Each turn includes its model requests, any explicitly assumed fallback, and one tool fee when applicable. Stacked bars separate Jev, direct/answer LLM calls, fallback LLM calls and tools. The line begins at zero and includes all these categories. Playback reveals only completed turns; axes remain fixed to the full planned run. Selecting a completed turn links to the existing canvas and request details without scrolling. Missing tariffs suppress the plot rather than showing a zero cost; valid zero tariffs remain zero. Token counts and prices remain assumptions. The x-axis measures turns, not elapsed time; no probability distribution or latency measurement is implied. ## Three distinct evidence layers 1. **Accounting correctness:** independent closed-form tests across counts, result sizes and history windows; component totals, fallback charges and context limits. These are software checks, not model experiments. 2. **Content and trajectory diagnostics:** real Jev/target runs in the same pinned Pi controlled world; compare completion, turns, tools and usage-priced cost. Known text allows retrospective token-error diagnosis. This does not validate predicted tool-output lengths or the new local draft generator. 3. **Prospective planning accuracy:** freeze a draft, its budgets and model configuration *before* running the target agent. Evaluate initial drafts and user-reviewed drafts separately. This layer has not yet been established. ## Existing evidence and the next matrix Run 35945790892 (code bdf7203f386f258c8a962d5e05972d9a37b9f763) completed 8 held-out tasks on both policies, with full usage coverage. Raw median cost error was 4.2%, calibrated median 1.4%, but the long task was overestimated by 26.6% raw and about 33.4% calibrated. These are content/trajectory diagnostics in two synthetic families, not an arbitrary-task accuracy claim. The new `matrix` suite contains 32 calibration + 32 held-out synthetic research tasks: - Four content formats: English prose, Chinese prose, JSON and code. - Two content lengths. - Two document-count settings per split (2/4 calibration; 3/6 held-out). - Clean reads and deterministic tool-retry cases. Bodies contain varied records/sections instead of one sentence repeated 100 times. Calibration and holdout use disjoint IDs and source bodies but the same world and generation rules. This remains in-domain, synthetic evidence. Matrix live results are pending. Offline demo success only checks wiring. Both policies use official Pi Agent 0.87.1, fixture-bound arguments, serial calls and terminal tools. No final prose generation. Keep the pre-existing smoke/standard/fresh suites for comparison. Matrix has at most 1536 decisions / 4608 request attempts, including bounded HTTP retries; live modes require explicitly authorized paid calls and verified prices. Report per format and length: completion, comparable-pair count, raw/calibrated median error, worst error and signed total-budget error. Report failures, unknown usage and per-task costs alongside averages. Do not claim broad accuracy from a median or from a large number of near-duplicate cases. ## Prospective study required before a broad accuracy article Freeze task families, split, target model, runner, tools, pricing, draft generator and evaluation rules before collection. Include document analysis, search/research, data processing, coding and multi-tool business tasks. Evaluate text types and input scales independently. Hold out whole task families as well as instances; repeating a fixture is not a new domain. Pilot to diagnose failures, then keep a final untouched test set. For each task save: task/environment description, pre-run exported assumptions, generator/user edits, predicted bill, actual trace and provider usage. Separate (a) mistaken execution assumptions, (b) token conversion, (c) tariff/cache mismatch, and (d) transport failures. Report completion and failures without replacing failed runs with successful retries. Model runs and any manual revisions must be distinguishable. An oracle analysis that uses observed outputs cannot be presented as a pre-run prediction. An article should link immutable code/run IDs and publish the protocol, per-task metrics, error distribution and large failures. The intended conclusion is conditional: under these declared assumptions and tested workloads, these errors were observed. Publication is driven by evidence, not a preselected savings or accuracy claim.