Cost engineering

Agentic delivery that survives the bill.

Token prices collapsed and AI budgets grew anyway, because an agent is a loop that re-reads its context on every turn. We compress that context, layer the memory behind it, and budget the path on cost per successful outcome.

compresscacherouteright-sizegovern

Why now

The cheap era was a pricing decision.

Per-token prices really did fall — roughly 280-fold for GPT-3.5-level inference in under two years. Enterprise bills went up anyway, because cheap tokens made agentic workloads viable and agentic workloads consume orders of magnitude more of them.

The subsidy was a pilot budget

Vendors absorbed GPU and token costs to win adoption. Unmetered, complimentary and included were pricing strategies, not permanent states.

Per-seat pricing hid per-token reality

A flat seat licence concealed a workload whose cost varies by an order of magnitude with the context each request carries.

Prototype defaults became architecture

Frontier model, full-context prompt, tool call every turn. Right for a demo, never revisited before the pilot was promoted.

≈280×

Fall in per-token price, GPT-3.5 level, 2022–2024

10 : 1

Typical prompt-to-output token ratio in enterprise workloads

≈1,000×

Tokens used by a coding agent versus code chat, per task

≈30×

Cost variance across repeat runs of the same task

Fig. 1Anatomy of agent spend. Input dominates output by roughly ten to one, and the input is re-read on every iteration of the loop — which is why cost scales with how much context you carry, not with how much code you ship. Click to enlarge.

An agent is a loop around a model

A chat completion sends its context once. An agent sends it again after every tool result, re-plan and retry — and most of that context is retrieved material and accumulated history, not the question. The lever with real leverage is not the price per token. It is how much the loop insists on carrying.

Framing draws on Harvard Business Review, Snowflake and Microsoft Azure. Price figures originate with the Stanford HAI AI Index. Full credits on our research page.

Measurement

Pick the unit before you pick the lever.

Optimise cost per call and you can halve spend, double the failure rate, and still show a win on the dashboard. The unit has to include whether the work got done.

Infrastructure health

Cost per 1,000 calls

Catches regressions in batching, cache hit rate and serving efficiency. An engineering metric, not a business one.

Capacity planning

Cost per agent run

The unit a path can be budgeted against — reported as a distribution, because variance across identical tasks is the norm.

The number that decides

Cost per successful outcome

Spend divided by runs that passed the gates. A cheaper run that fails is the most expensive thing in the system.

Six places the money sits

Inference is the only line most teams can see, and rarely the largest.

Model servingInference per token or per provisioned unit.

Training & tuningUp-front spend that buys a permanently smaller runtime bill.

Hosting & networkOrchestration, queues, retries, sandboxes, egress.

Context storageVector and graph stores, embeddings, indexing jobs.

Application layerHarness, tool gateway, evals, observability pipeline.

Operational supportHuman review, escalation, incident response, governance.

Cost-component structure adapted from Google Cloud and its FinOps for generative AI guidance.

paths/feature-delivery.cost.yamlyaml

budget:
  unit: cost_per_successful_run    # not per call, not per token
  target: 1.00
  ceiling: 2.50                    # path fails closed above this

context:
  max_input_tokens: 48000
  compressors: [select, abstract, densify]
  handoff: structured              # artefact, never transcript

memory:
  layers: [user, team, domain]
  read_order: narrowest_first

routing:
  default: small
  escalate_when: [low_confidence, gate_rejection]

cache:
  prompt: true                     # stable content first
  semantic: { enabled: true, similarity_threshold: 0.94 }

gates:
  quality_regression_tolerance: 0.02
  on_breach: revert_last_lever

Cost belongs in the path definition

A budget in a finance dashboard gets reviewed monthly. A budget in the path definition gets enforced every run, and a breach arrives as a failing path with a named owner. It is the same file that defines the gates.

Fig. 2Cost governance as a loop, not a cleanup. Each pass moves one lever and re-measures quality, because the only number worth optimising is the cost of a successful outcome — a cheaper run that fails is the most expensive thing in the system. Click to enlarge.

The lever stack

Sequence matters more than technique.

Savings compose, and they compose in one direction. Compression shrinks what you cache, caching reduces what you route, routing changes which model is worth right-sizing. Run it backwards and you distil a model to serve context you should have deleted.

Fig. 3The lever stack. Applied in this order the levers compound rather than add: compression shrinks what you cache, caching reduces what you route, routing changes which model you right-size. Applied out of order, each one optimises work the previous one should have removed. Click to enlarge.

Model routing

Small model first, escalate on low confidence. Reported routers cut large-model calls by up to 40%; cascades up to 60% with under 1% quality loss.

Prompt & semantic caching

Stable content first, volatile last, so identical prefixes bill once. Semantic caching answers near-duplicate intent — around two thirds fewer upstream calls.

Continuous batching

Paged KV-cache scheduling has been reported at two to four times the throughput on the same hardware.

Quantisation

16-bit to 4-bit cuts memory roughly fourfold; activation-aware schemes protect the weights that carry accuracy.

Prefill optimisation

Long agent contexts inflate prefill. Published work reports about half the prefill compute at double the throughput.

Commitment & placement

Predictable volume on provisioned throughput, latency-tolerant work on batch endpoints at roughly half the price.

Published figures belong to their original authors and are summarised for orientation, not as guarantees. Technique inventory from Snowflake's research survey and Azure's runtime levers. Measure on your own workload.

Context compressors

Send less, not worse.

Input dominates the bill and the loop pays for it again every turn. A compressor reduces what the model reads while preserving what it needs — which is why each one ships with an eval attached.

Fig. 4The context compressor chain. Each stage is cheaper than the one before it and lossier than the one before it, so every stage is paired with an evaluation that measures task success rather than token count. A compressor that saves tokens and loses the answer has not saved anything. Click to enlarge.

select

Relevance selection

Retrieve against the bounded context, dedupe, drop boilerplate. The cheapest token is the one you never send.

largest single reduction, and lossless

abstract

Hierarchical summarisation

Transcripts and file history collapse into rolling digests with drill-down. The original is fetched only when the digest falls short.

turns history growth into a conditional cost

densify

Token-level compression

Learned compressors strip low-information tokens from what must still be read closely. LLMLingua-class results report up to twentyfold.

up to 20× on the passages it touches

structure

Structured handoff

Agents pass artefacts, not conversations. One published domain-specific encoding reports 31% fewer input tokens and 34–63% lower latency.

removes context re-establishment per handoff

constrain

Schema-constrained output

Output tokens are generated one forward pass each. A tight response schema deletes preamble nobody parses.

shortens decode, which dominates latency

Every compressor is a bet

The bet is that the discarded tokens did not matter. It is usually right and occasionally catastrophic, and the failure mode is not an error — it is a confident answer built on pruned material. So a stage ships behind an eval on the path's golden set, with a tolerance and an automatic revert.

Multi-layer memory

Recall is cheaper than re-reading.

Most context is not new information — it is the same information, resent. Layering memory by scope turns that repetition into a lookup, and the narrowest layer that can answer is the only one that gets read.

Fig. 5Multi-layer memory. Separating memory by scope is a cost decision as much as a correctness one: recall is cheaper than re-reading, the narrowest layer that can answer is the one that gets read, and consolidation happens on a schedule rather than on every turn. Click to enlarge.

User memory

scope: one personttl: rolling

Preferences, recurring intent, how this engineer works.

Removes the orientation paragraph that opens every session.

Team memory

scope: one squadttl: release cycles

Conventions, review standards, decisions still in flight.

Stops five engineers paying to rediscover the same convention.

Domain memory

scope: bounded contextttl: versioned

Ontology, specs, ADRs, incidents — provenance on every fact.

Replaces broad retrieval with precise recall.

Four rules that keep layers from becoming cost

Narrowest layer first

Stop at the first layer that answers. Reading all three every turn recreates the problem the layers exist to solve.

Write on a schedule, not per turn

The Context Curator promotes what recurred and expires what went stale. Per-turn writes build a second transcript.

Memory is a security boundary

Layer reads are scoped by identity at the tool gateway. Cross-layer promotion is explicit and audited.

Provenance or it does not persist

A fact without a source cannot be corrected when it turns out to be wrong.

The context engines Memory as attack surface

memory.policy.yamlyaml

layers:
  user:
    engine: supermemory
    container: user/{subject}
    ttl: 90d
    read: [self]
    budget_tokens: 1200

  team:
    engine: supermemory
    container: team/{squad}
    ttl: 2_release_cycles
    read: [squad, squad_agents]
    budget_tokens: 4000

  domain:
    engine: cognee + hindsight
    container: domain/{bounded_context}
    ttl: versioned            # expires by release, not by clock
    read: [domain_agents]
    requires_provenance: true
    budget_tokens: 18000

assembly:
  order: narrowest_first
  stop_when: sufficient
  total_budget_tokens: 24000

consolidation:
  owner: phaiai/agents/context-curator
  schedule: nightly
  promote_after: 3_recurrences

The Domain Context Engine, priced

The domain layer is the same engine our agents are already wired into, built on Cognee, Supermemory and Hindsight. The cost lens adds a token budget per layer and a read order — which turns “the agent has memory” into a number you can put in a path definition.

How we engage

A cost programme, not a cost cut.

We instrument what you actually spend, sequence the levers against your own workload, and leave the measurement behind — because this quarter's levers will be wrong the moment your model mix changes.

012 weeks

Baseline

Tag every model and tool call. Produce cost per run and cost per successful outcome, per path. No changes yet.

023–4 weeks

Sequence

Compress, then cache, then route, then right-size. One lever at a time, each measured before the next is allowed.

032 weeks

Gate

Budgets and quality gates move into the path definitions, so a regression fails the path instead of the invoice.

04ongoing

Operate

A FinOps practice for agentic spend: allocation by domain, forecast against roadmap, standing review.

Know what a successful outcome costs you today.

Most teams can say what last month's AI bill was and cannot say what one delivered feature cost. The baseline engagement answers the second question, which makes every lever after it obvious.

Book a cost baseline Sources and credits