When is LLM
fallback worth it?

The question behind the experiment

We built jev-agent-router around a straightforward idea: ask Jev first, accept confident answers, and call an LLM when the router needs a fallback.

Then we tested whether the extra call was worth making.

On our revised BANKING77 development split, Jev alone was both cheaper and more accurate than every cascade threshold we evaluated. Our original selector still recommended a cascade, because it could only choose between cascade thresholds.

That was a problem with the decision we had allowed the system to make. We updated the selector to compare Jev-only, LLM-only, and cascade. Reanalyzing the same development records selected Jev-only.

This case study explains the experiment, a category-description issue we found along the way, and what the results do—and do not—establish about fallback.

A bounded task, a reproducible setup

We used BANKING77, an English banking-intent dataset with 77 categories. This measures bounded intent classification, not complete agent execution or tool-use success.

The experiment used 20 training examples for a smoke check, 385 for development, and 770 test examples—ten per category. Smoke and development examples were disjoint. Six exact text overlaps between the source training and test files were excluded from the test pool before sampling. All requests included all 77 candidate categories.

We compared Jev jev-latest with gemini-3.1-flash-lite through Google's Gemini service, using JSON Schema output for the LLM. Calls were sequential on GitHub Actions. The revised data collection ran on September 22, 2026; its workflow run records the source commit and execution settings.

First, define the task correctly

Before interpreting the results, we had to check what we were asking the models to classify.

Our first implementation generated category descriptions by replacing underscores in each label with spaces. That seemed reasonable until we inspected get_physical_card: its examples in the pinned source data were about finding or receiving a card PIN. Requests to order an actual physical card belonged to order_physical_card.

We checked the source-file hashes and compared every sampled text and label against the original CSVs. The mapping was preserved correctly. The misleading instruction came from treating the label name as a reliable definition.

We replaced these descriptions with versioned English definitions grounded in training examples. Both models received the same definitions; original labels and sampled examples stayed unchanged. The review notes document the method, including its limits: this was a sampled review across all categories, not an exhaustive annotation audit.

On the same test subset, Jev accuracy rose from 82.08% to 87.53%, and Gemini from 81.17% to 86.36%. Both went from zero to nine correct answers out of ten in the PIN category. These observations support clearer task definitions, but they are not a controlled estimate of the description change alone: the models were called again, and their outputs can vary.

The definitions also increased input length. Jev's estimated cost rose from $0.0719 to $0.1083 per thousand requests. Better instructions have a cost that belongs in the comparison.

Give the selector a single-model option

For strategy selection, our rule was to choose the lowest-cost candidate whose observed development accuracy was no more than one percentage point below the LLM baseline. That is an experiment setting, not a statistical guarantee of equivalent quality.

Here is the development comparison after the description revision:

The other tested thresholds—0.6, 0.7, 0.8, 0.9, and 0.95—also cost more and scored below Jev-only on development. The updated selector therefore chose Jev-only. On these records, it cost about 78% less than the Gemini baseline at the supplied tariffs. This is a development-set estimate, not a claim of universal savings.

For reproducibility, the run's configured USD-per-million-token rates were Jev input $0.042 and output $0, and Gemini input $0.25 and output $1.50. These are the experiment's supplied tariffs, not a live pricing quote. Estimates exclude cached-input discounts, free credits, and billing adjustments.

What happened on the test subset?

We then applied the development-selected strategy to the existing paired test records. We kept the development-selected cascade threshold of 0.5 as a diagnostic comparator:

The cascade scored three more correct answers on test. We did not change the selected strategy to chase that result: selection used development records.

Gemini had one failed test request without token usage. It remains in the accuracy denominator, and its unknown cost prevents a complete test-set LLM cost total. We do not treat that request as free or publish an exact test-set savings percentage against Gemini. Cascade replay still has complete cost coverage because it did not use that missing-usage LLM result.

Replay estimates. Live measurements.

The cascade figures above are replay, which means each test request had already been sent independently to both models. Offline analysis chooses the stored Jev answer above the threshold and the stored LLM answer when fallback applies. It estimates what that policy would have spent; the original paired collection actually paid for both calls on every request.

We also measured a real cascade at threshold 0.5 during the original revised run:

The real cascade made 16 fallback calls, compared with 14 in replay. Repeated calls can produce different answers and confidence scores. These two executions had the same number of correct answers, which does not mean identical predictions. The measured cascade cost approximately 9.4% more than Jev-only in this comparison.

Latency depends on the runner, network, provider conditions, and collection phase. Replay has no measured end-to-end cascade latency. The live cascade above used its original policy; the later Jev-only selection was an offline reanalysis, not a new live deployment experiment.

Did confidence pick useful requests?

To examine whether confidence was choosing useful requests, we added a random fallback control. For the test comparator, the confidence rule selected 14 fallback calls. Those calls corrected four Jev errors and replaced one correct Jev answer with an incorrect LLM answer: a net gain of three.

What if we had picked 14 requests at random instead?

The report runs 10,000 deterministic simulations with seed 42. The control matches fallback call count, not token cost. Recoverable Jev errors always fall back, nonrecoverable request errors never do, and randomization applies to valid Jev responses. Failed LLM outputs remain failures.

On these test records, confidence selected more useful requests than random allocation on average. However, the development results tell a less favorable story: 16 confidence-based fallbacks corrected one error and introduced three, for a net loss of two.

Both observations matter. Random simulations reuse fixed outputs; they do not create new independent model experiments or establish that the same benefit will generalize.

What this experiment cannot tell us

There are further limits to this case study. We examined one public task, one model pair, and one sampled test subset. jev-latest is a moving alias. The test results had already been inspected before the category-description and selector revisions, so this is a retrospective revised evaluation, not untouched held-out validation. Small accuracy differences should not be presented as proof of superiority or equivalence.

A decision layer that can choose less

The practical change is already in the open-source benchmark: it can now recommend either single provider or a cascade, freeze that choice, and report fallback corrections, regressions, and a random control. Existing paired records can be reanalyzed without more API calls; the reanalysis instructions describe the process.

For JevCalc, this gives us a concrete direction: help developers decide whether an extra model call earns its cost on their own workload. In this experiment, the development evidence favored Jev alone. A useful decision layer needs to be able to make that recommendation—and keep measuring when the workload changes.

Dataset attribution: BANKING77 is distributed by PolyAI under CC-BY-4.0. See Casanueva et al., *Efficient Intent Detection with Dual Sentence Encoders*, 2020.