How to Control LLM Costs in a Startup: Caps, Caching and Fallback Ladders

To control LLM costs in a startup, make every model call counted, capped and cached where the input repeats, and route each call to the cheapest model that can do the job with a fallback ladder behind it. Those four verbs (count, cap, cache, route) are the whole discipline; everything else in this article is the detail of doing them without breaking the product.

We run fifteen AI agents for early-stage founders on a cost base that has to stay close to flat while usage moves. The rules below came out of a spend audit we ran on our own pipeline, and each one is now enforced by a test rather than by memory.

Count every model call before you try to cap it

You cannot cap what you cannot count. The first thing the audit found was not a runaway bill but a set of paths where nobody could say how many calls a single user action produced. A pitch-deck generation, a research run with a background extractor, a retry after a timeout: each looked like one call and each was several.

Every model call in our stack now goes through one service, and every path through that service has a test that asserts its call count. When we add a model call, we write the test that asserts how many times it fires. That single habit has caught more waste than any dashboard, because a dashboard shows you the bill after the month and a test shows you the extra call before the merge.

Count at the boundary you can act on. For us that is per organization per day, because that is where a cap can be enforced and where a founder would recognise the number. Per-request counts feed into it; per-token counts are useful for pricing but too fine to gate on.

Give every caller a budget floor, not an unlimited default

An organization with no explicit budget row used to get unlimited tool calls. That is the default in most systems, because "no limit configured" reads as "no limit," and it is the wrong default for anything that spends money on someone else's behalf. An org with no budget now gets a daily floor of 500 tool calls, checked and reserved atomically before the call is made, so two concurrent requests cannot both squeeze through the last slot.

The floor is deliberately generous. It is not a pricing lever; it is a circuit breaker. A founder running 128 agent invocations in a week (the deepest single-founder engagement we have seen) never came near it. A loop bug, a runaway integration or a scripted abuser hits it in minutes. Those are the cases a floor exists for.

Put the same shape on every autonomous job. Our background crons park a prospect after three failed drafts and stop re-reviewing a draft after three unjudged attempts, because a job that retries forever is a budget with no ceiling. And one environment variable on the cron service stops every model-calling job at once, which is the lever you want at 2 a.m. We cover where that switch has to live in human-in-the-loop AI governance for startups.

Route to the cheapest model first and put a fallback ladder behind it

Most of what an agent does is not hard. Extracting facts from a research result, classifying a question, rewriting a paragraph in house style: a small fast model handles these at a fraction of the cost of the reasoning model. Our default and fast tier is the lightest model in the family; a heavier model exists only as the fallback rung.

The fallback matters because the cheap model is not free of failure modes. Ours degenerates roughly one call in three on very heavy prompts, so every agent call carries a tripwire on output size: if the response blows past the ceiling, the call is treated as failed and the ladder steps to the next model. The same happens on a provider 5xx, a safety block or an empty response.

Two rules keep the ladder from becoming its own cost problem:

• A 400, 403 or 404 from the provider is definitive. The request was malformed, forbidden or aimed at a model that does not exist. Sending it to the fallback model buys a second bill for the same error.
• The ladder never sends the same prompt to the same model twice. Retrying an identical input against an identical model is a coin flip you are paying for.

The consequence is that an agent invoke costs at most two model calls, and a test asserts that ceiling.

Run extractors on the cheapest model with a length guard

The feature that makes our product work is Shared Context: every agent's output is written back into one per-company memory, and every later agent reads from it. The writing half is done by extractors, background calls that pull structured facts out of an agent's answer. They are the easiest place to double your bill without noticing, because they run on every response and nobody watches them.

Three constraints keep them cheap. They run on the fast model only. They carry a length guard, so a very long answer is truncated before extraction rather than shipped whole. And they run through a bounded background runner, so a burst of agent traffic queues extraction rather than fanning it out. Extraction is fire-and-forget behind a broad exception handler, and its outcome is stamped on the interaction log so a health command can report attempted-versus-produced per agent.

That last point is worth a sentence. A background call that fails silently costs nothing, which sounds like a saving until you realise the product feature it powered has been off for weeks. Cost control and correctness share the same instrument: count what fired, count what it produced, compare.

Cache anything deterministic, and stop generating what a library can serve

A prompt whose inputs repeat should never reach the model twice. Our technical documentation autofill, roadmap resource suggestions and trending-topic lookups all go through a result cache keyed on the input, because the same company asking the same question on Tuesday and Wednesday should pay once.

The bigger saving came from asking whether a call needed a model at all. Every pitch deck we generate carries a visual per slide. We had been generating those with an image model on each deck, a 4-to-10 second, quota-bound call per slide. The audit showed the prompt was always a per-slide template with the founder's content explicitly excluded, so each call was an expensive re-roll of a stock picture. We replaced it with a committed library of 20 slides by 4 variants, chosen by hashing the organization id with the slide name. The model now runs only when a founder clicks "Generate unique visual." Same decks, near-zero image spend.

The general test: if you can predict the output category before the call, you do not need the call.

Stop the frontend from paying twice

Half our wasted calls came from the browser, not the backend. Three patterns caused nearly all of them, and each is a one-line fix once you know to look.

Effects that fire twice. React development mode double-mounts components, so an effect that triggers an AI call on mount fires two requests. Anything that starts a model call from an effect must check an in-flight reference synchronously before firing.

Exports that walk the product. A deck export that steps through every slide with the generation effects still live re-triggers generation on each step. Exports must render from persisted state with the enqueue effects disabled.

Results that are not saved. A generated result that lives only in component state is regenerated on the next open. Every generated result is persisted to agent state, because a founder reopening a deck should pay nothing.

None of these show up in a backend cost report as anything but "usage went up." Read the frontend for the cause before you touch the model routing.

A cost audit you can run this week

Do this in order; the early steps make the later ones measurable.

1. Route every model call through one function and log a length, a model name and a caller. Never log the prompt text.
2. Write a call-count test for your three most-used paths. The number it asserts is usually a surprise.
3. Set a daily floor per tenant where "no budget configured" currently means unlimited.
4. Add the definitive-error rule so a 4xx never reaches a fallback model.
5. List every prompt whose inputs repeat and put a cache in front of it.
6. Grep the frontend for effects that call the API and check each one for an in-flight guard.

Where the calls go once they are counted, and how fifteen agents share one memory without stepping on each other, is in multi-agent system design lessons. If you are choosing what to buy rather than build, the agent library and pricing show what a founder pays for a system where these controls already exist.

Frequently Asked Questions

What is the fastest way to reduce LLM costs in a startup?

Count calls per user action first, then cap per tenant per day. Most teams find one path making three to five calls where they assumed one. A call-count test on each path catches the extra calls before they ship.

Should a startup use a cheaper model by default?

Yes. Route to the lightest model that handles the task and keep a heavier one as a fallback rung. Put a tripwire on output size so a degenerate response fails fast, and never send a prompt that returned a 400, 403 or 404 to the fallback.

How much does caching save on LLM costs?

It depends on how often inputs repeat. Deterministic prompts (autofill, suggestions, lookups) should reach the model once per distinct input. Our largest single saving was not a cache but a stock library that replaced a per-slide image call on every pitch deck.

What is a budget floor for AI calls?

A default daily cap applied when a tenant has no explicit budget configured. Ours is 500 tool calls per organization per day, reserved atomically before each call. It is a circuit breaker for loop bugs and abuse, not a pricing tier.

Why do frontend bugs increase LLM costs?

Because the browser can request the same generation several times: development mode double-mounts effects, exports step through slides with generation live, and unsaved results regenerate on every open. Guard effects with an in-flight reference and persist every generated result.