---
title: "When Is LLM Fallback Worth It? A Jev Routing Experiment on BANKING77"
description: "We compared Jev, Gemini, and a confidence-gated cascade. The results led us to let our router recommend a single model—and measure whether fallback earns its cost."
slug: "jev-banking77-fallback-routing-experiment"
draft: false
---

# When Is LLM Fallback Worth It? A Jev Routing Experiment on BANKING77

We built [jev-agent-router](https://github.com/luhayes/jev-agent-router) around a straightforward idea: ask Jev first, accept confident answers, and call an LLM when the router needs a fallback.

Then we tested whether the extra call was worth making.

On our revised BANKING77 development split, Jev alone was both cheaper and more accurate than every cascade threshold we evaluated. Our original selector still recommended a cascade, because it could only choose between cascade thresholds.

That was a problem with the decision we had allowed the system to make. We updated the selector to compare **Jev-only, LLM-only, and cascade**. Reanalyzing the same development records selected Jev-only.

This case study explains the experiment, a category-description issue we found along the way, and what the results do—and do not—establish about fallback.

We used [BANKING77](https://github.com/PolyAI-LDN/task-specific-datasets/tree/57ec275d8078af65b7731c2a98be812d844a6d6b/banking_data), an English banking-intent dataset with 77 categories. This measures bounded intent classification, not complete agent execution or tool-use success.

The experiment used 20 training examples for a smoke check, 385 for development, and 770 test examples—ten per category. Smoke and development examples were disjoint. Six exact text overlaps between the source training and test files were excluded from the test pool before sampling. All requests included all 77 candidate categories.

We compared Jev `jev-latest` with `gemini-3.1-flash-lite` through Google's Gemini service, using JSON Schema output for the LLM. Calls were sequential on GitHub Actions. The revised data collection ran on September 22, 2026; its [workflow run](https://github.com/luhayes/jev-agent-router/actions/runs/35714478131) records the source commit and execution settings.

Before interpreting the results, we had to check what we were asking the models to classify.

Our first implementation generated category descriptions by replacing underscores in each label with spaces. That seemed reasonable until we inspected `get_physical_card`: its examples in the pinned source data were about finding or receiving a **card PIN**. Requests to order an actual physical card belonged to `order_physical_card`.

We checked the source-file hashes and compared every sampled text and label against the original CSVs. The mapping was preserved correctly. The misleading instruction came from treating the label name as a reliable definition.

We replaced these descriptions with versioned English definitions grounded in training examples. Both models received the same definitions; original labels and sampled examples stayed unchanged. The [review notes](https://github.com/luhayes/jev-agent-router/blob/a60285ab70e471d0b473952a729c5f94f7d1f452/benchmarks/banking77/CRITERIA.md) document the method, including its limits: this was a sampled review across all categories, not an exhaustive annotation audit.

On the same test subset, Jev accuracy rose from 82.08% to 87.53%, and Gemini from 81.17% to 86.36%. Both went from zero to nine correct answers out of ten in the PIN category. These observations support clearer task definitions, but they are not a controlled estimate of the description change alone: the models were called again, and their outputs can vary.

The definitions also increased input length. Jev's estimated cost rose from $0.0719 to $0.1083 per thousand requests. Better instructions have a cost that belongs in the comparison.

For strategy selection, our rule was to choose the lowest-cost candidate whose observed development accuracy was no more than one percentage point below the LLM baseline. That is an experiment setting, not a statistical guarantee of equivalent quality.

Here is the development comparison after the description revision:

| Strategy | Correct / 385 | Accuracy | Estimated USD / 1,000 requests |
|---|---:|---:|---:|
| Gemini-only | 336 | 87.27% | $0.4921 |
| Jev-only | 341 | 88.57% | $0.1083 |
| Cascade, threshold 0.5, replay | 339 | 88.05% | $0.1288 |

The other tested thresholds—0.6, 0.7, 0.8, 0.9, and 0.95—also cost more and scored below Jev-only on development. The updated selector therefore chose **Jev-only**. On these records, it cost about 78% less than the Gemini baseline at the supplied tariffs. This is a development-set estimate, not a claim of universal savings.

For reproducibility, the run's configured USD-per-million-token rates were Jev input $0.042 and output $0, and Gemini input $0.25 and output $1.50. These are the experiment's supplied tariffs, not a live pricing quote. Estimates exclude cached-input discounts, free credits, and billing adjustments.

We then applied the development-selected strategy to the existing paired test records. We kept the development-selected cascade threshold of 0.5 as a diagnostic comparator:

| Strategy | Correct / 770 | Accuracy | Estimated USD / 1,000 requests |
|---|---:|---:|---:|
| Gemini-only | 665 | 86.36% | Unknown: one request lacks usage |
| **Jev-only, selected** | **674** | **87.53%** | **$0.1083** |
| Cascade replay, diagnostic comparator | 677 | 87.92% | $0.1172 |

The cascade scored three more correct answers on test. We did not change the selected strategy to chase that result: selection used development records.

Gemini had one failed test request without token usage. It remains in the accuracy denominator, and its unknown cost prevents a complete test-set LLM cost total. We do not treat that request as free or publish an exact test-set savings percentage against Gemini. Cascade replay still has complete cost coverage because it did not use that missing-usage LLM result.

The cascade figures above are **replay**, which means each test request had already been sent independently to both models. Offline analysis chooses the stored Jev answer above the threshold and the stored LLM answer when fallback applies. It estimates what that policy would have spent; the original paired collection actually paid for both calls on every request.

We also measured a real cascade at threshold 0.5 during the original revised run:

| Measured execution | Accuracy | Estimated USD / 1,000 requests | P50 | P95 |
|---|---:|---:|---:|---:|
| Jev-only calls in paired collection | 87.53% | $0.1083 | 105 ms | 158 ms |
| Actual cascade, separate collection | 87.53% | $0.1185 | 106 ms | 184 ms |

The real cascade made 16 fallback calls, compared with 14 in replay. Repeated calls can produce different answers and confidence scores. These two executions had the same number of correct answers, which does not mean identical predictions. The measured cascade cost approximately 9.4% more than Jev-only in this comparison.

Latency depends on the runner, network, provider conditions, and collection phase. Replay has no measured end-to-end cascade latency. The live cascade above used its original policy; the later Jev-only selection was an offline reanalysis, not a new live deployment experiment.

To examine whether confidence was choosing useful requests, we added a random fallback control. For the test comparator, the confidence rule selected 14 fallback calls. Those calls corrected four Jev errors and replaced one correct Jev answer with an incorrect LLM answer: a **net gain of three**.

What if we had picked 14 requests at random instead?

| Fixed test records, 14 fallback calls | Net correct-answer gain over Jev |
|---|---:|
| Confidence-based selection | +3 |
| Random selection, expected gain | −0.16 |

The report runs 10,000 deterministic simulations with seed 42. The control matches fallback **call count**, not token cost. Recoverable Jev errors always fall back, nonrecoverable request errors never do, and randomization applies to valid Jev responses. Failed LLM outputs remain failures.

On these test records, confidence selected more useful requests than random allocation on average. However, the development results tell a less favorable story: 16 confidence-based fallbacks corrected one error and introduced three, for a net loss of two.

Both observations matter. Random simulations reuse fixed outputs; they do not create new independent model experiments or establish that the same benefit will generalize.

There are further limits to this case study. We examined one public task, one model pair, and one sampled test subset. `jev-latest` is a moving alias. The test results had already been inspected before the category-description and selector revisions, so this is a **retrospective revised evaluation**, not untouched held-out validation. Small accuracy differences should not be presented as proof of superiority or equivalence.

## Why use jev-agent-router?

The conclusion is not that every application needs a cascade. It is that developers need a repeatable way to decide whether a second call earns its cost, and to execute that decision consistently. On this development set, that meant Jev-only.

**jev-agent-router packages two distinct capabilities:** evaluation tools that compare Jev-only, LLM-only and cascade, and a runtime Router that validates Jev responses and bounds fallback attempts. The selected-policy application example also dispatches either single-provider strategy. JevCalc publishes the evidence and provides cost exploration and optional telemetry; no JevCalc account is required to use the SDK.

| What this experiment exposed | What you can reuse |
| --- | --- |
| A high confidence score can still be wrong | Evaluate thresholds against your labeled requests; inspect accepted errors. |
| Fallbacks can correct and regress | Count both outcomes and compare against matched-call random controls. |
| A single provider can win | Select among all three strategies on dev, then freeze the choice. |
| Replay has no measured cascade latency | Execute the frozen policy in a separate live measurement. |
| Failure paths need explicit behavior | Reuse local validation, bounded calls, cancellation and observer diagnostics. |

Adopting the package gives your team a shared set of reviewable artifacts: a versioned dataset manifest, comparable reports, a frozen policy and a failure-behavior report. The application example carries that evaluated configuration into execution. You can inspect how a choice was made, reproduce its checks and repeat the process when the workload changes.

## Run the same process on your own workload

The [decision workflow](https://github.com/luhayes/jev-agent-router/blob/main/docs/decision-workflow.md) takes you from labeled requests to an evaluated policy. Start with the offline demo, then import your own explicit smoke, development and test splits. The importer validates labels, detects duplicate IDs and normalized exact-text overlap, and records integrity hashes. It cannot detect every form of semantic or customer-level leakage.

| Stage | Artifact and review |
| --- | --- |
| Define | Versioned criteria and a decision record with quality, failure, latency and cost limits. |
| Collect | Small smoke check, then paired development calls with supplied prices. |
| Choose | Jev-only, LLM-only and cascade comparison; freeze policy.json on dev. |
| Validate | Evaluate the frozen policy on untouched test data; inspect random controls and class errors. |
| Measure | Make a separate live run of only the selected strategy for wall-clock latency. |
| Integrate | Load the same manifest and policy in the application example; handle errors explicitly. |
| Observe | Run fault checks, begin a limited rollout and maintain labeled quality review. |

The current selector enforces a development accuracy-drop constraint while minimizing fully known estimated cost. Absolute accuracy, P95, request-failure and critical-class requirements remain explicit review gates in the decision record. A selected policy is not automatic permission to release.

Follow [the copyable commands](https://github.com/luhayes/jev-agent-router/blob/main/docs/decision-workflow.md), then [integrate the selected policy](https://github.com/luhayes/jev-agent-router/blob/main/examples/selected_policy.py). The demo, import, analysis and fault experiment are offline. Real collection and application inference are paid calls that send inputs to the configured providers. Private workloads should stay out of public repository artifacts.

## What happens when a provider fails?

We also ran a separate offline fault experiment through the actual Router and JSON fallback parser. It covers 17 synthetic scenarios, repeated three times: 99 executions and **693 of 693 behavior checks passed**. The control uses the same validated Router at threshold zero without fallback; the treatment uses threshold 0.8 with fallback. This isolates policy behavior, not an advantage over every possible custom router.

| Injected condition | Observed behavior |
| --- | --- |
| 429, 503, timeout, network failure, malformed JSON, unknown label or invalid probability sum | Jev-only abstained; cascade returned the valid fallback fixture. |
| HTTP 401 | Both stopped; no fallback call hid the authentication failure. |
| Fallback timeout, HTTP 503 or invalid label | Cascade returned an explicit error; it did not invent a valid answer. |
| Cancellation during Jev or fallback | Cancellation propagated; attempts remained bounded. |
| Confidently wrong answer | Both accepted the wrong Jev fixture. |
| A wrong fallback after a correct low-confidence Jev answer | The cascade introduced an error. |

**A contract PASS can be an expected error, not a successful inference.** These hand-picked cases do not measure production availability or model accuracy. Mocked request counts are not billable calls. Local timing includes intentional 50 ms timeouts and is not a provider-latency comparison. The fallback-stage cancellation case applies only to cascade, which explains the different execution counts between groups.

[Reproduce the fault experiment](https://github.com/luhayes/jev-agent-router/blob/main/benchmarks/faults/README.md) without API keys, or download [the full observed report](/research/router-fault-report.json). The report preserves all repeated outcomes, expected contracts, diagnostics and source hashes. This establishes inspectable behavior for tested failure paths; production quality still needs representative labels and monitoring.

Dataset attribution: BANKING77 is distributed by PolyAI under CC-BY-4.0. See Casanueva et al., [*Efficient Intent Detection with Dual Sentence Encoders*](https://arxiv.org/abs/2003.04807), 2020.