Abstract
Conversational APIs commonly submit the complete working context at every turn, but a modern inference engine can reuse an exact cached prefix rather than evaluate every token again. Eva explores what remains to gain when the alternative already handles that cache well. It keeps the causal sequence active and sends only the new information. In a controlled 16-turn Qwen3-4B conversation, stateful suffix append reached the first token in 131.8 milliseconds on average, versus 175.6 milliseconds with ideal same-worker prefix reuse—a 1.33× speed ratio, or about 30% faster. This note explains the result and its boundaries.
What remains after prompt caching?
An agent can remember a conversation at the application level while still submitting the complete transcript, instructions, tools, and retrieved memory on every request. With a matching prefix cache, the engine need not evaluate all of those tokens again.
A hit is not necessarily free. The system still receives and tokenizes the request, finds and verifies the prefix, reconciles any divergent suffix, and may route or restore cached KV. Stateful inference asks whether an already-active local sequence can remove some of that remaining boundary work.
Submitting the full transcript is not the same as recomputing it—but it is not the same as appending only the delta.
Why stateless inference became the default
Stateless inference is not simply a technical oversight. It fits the scalability needs of cloud providers. Independent requests can be routed across large fleets, balanced between workers, retried after failures, isolated between customers, and scaled up or down without keeping a specific session attached to a specific accelerator.
Resident inference state changes that equation. A live KV cache consumes memory and creates affinity between a conversation and the worker that holds it. At cloud scale, that complicates scheduling, capacity planning, failover, and fair use across many concurrent customers. Prefix caching recovers much of the computation while preserving a stateless API, although its reuse remains infrastructure-managed.
A local configuration relaxes much of that constraint. One user, one model process, and a small number of active conversations do not need hyperscale request routing. A lightweight session can remain attached to its local worker, retain its valid causal state, and append only what changed.
The cloud optimizes for interchangeable requests. A local agent can optimize for continuity.
Stateful inference in one sentence
Keep the model’s valid causal sequence and KV state resident between turns, then evaluate only the newly appended suffix.
This is not a claim that physical KV state replaces memory. Eva keeps an authoritative semantic event history separately. Resident KV can be reused, invalidated, rebuilt, or discarded; the transcript remains the source of truth.
What Eva keeps alive
Eva runs a long-lived native llama.cpp worker. The worker retains the model, the llama_context, the generated token sequence, and its KV cache. At each turn it renders the complete semantic message list for comparison, verifies the longest valid resident prefix, and decodes only the suffix.
A real wake phase prefills stable identity and continuity context before the first user message. If the causal prefix changes—or the context approaches its configured high-water mark—Eva rebuilds from bounded semantic history rather than pretending that stale KV is still valid.
What about prompt caching?
The KV cache used while generating one response does not automatically survive the next request. Cross-request prefix caching can go further: when the exact token prefix is still cached and the request reaches a compatible worker, it can reuse that computation and recover much of the prefill saving.
For normal operation, prefix reuse is the appropriate baseline. A cache hit can still require lookup, routing, allocation, or KV transfer; the cost depends on whether the prefix is already on the accelerator or must be restored. Our controlled baseline is deliberately favorable: an exact hit on the same worker, without distributed lookup or restoration.
What we measured
Across turns 2–16 of a controlled synthetic Qwen3-4B conversation, direct stateful append averaged 131.8 milliseconds to first token versus 175.6 milliseconds for ideal same-worker prefix-cached replay—about 30% faster.
The benchmark compares normal cache-aware operation directly. Prefix-cached replay submitted the complete transcript and reconciled its exact matching prefix against resident KV. Stateful append submitted only each new delta. Both modes ran on the same model and worker, and both passed five out of five retention checks.
| Qwen3-4B · 16 turns | Evaluated prompt tokens | Wire characters | Mean TTFT · turns 2–16 | Memory checks |
|---|---|---|---|---|
| Same-worker prefix reuse | 765 | 29,348 | 175.6 ms | 5 / 5 |
| Stateful suffix append | 588 | 1,389 | 131.8 ms | 5 / 5 |
Direct stateful append was 1.33× faster than this ideal same-worker cache hit and sent 95.3% fewer characters over the request boundary. Expressed as latency rather than speed, mean TTFT fell by 25%. This is one synthetic conversation, not a universal performance guarantee.
So, is it really 30%?
In this run, yes: 175.6 divided by 131.8 gives a 1.33× speed ratio, which we round to about 30% faster. The equivalent reduction in mean TTFT is 25%, because “percent faster” and “percent less latency” use different denominators.
The result will vary with the model, hardware, context length, new-message length, prompt-cache implementation, and whether the causal prefix remains valid. The claim is therefore specific to this controlled 16-turn experiment.
Four design rules
- Keep semantic history authoritative. KV is an optimization, not your database.
- Append only to a verified prefix. You cannot insert new facts into an already-decoded causal past.
- Make rebuilds visible. Report sequence resets, prompt tokens, TTFT, and context bounds rather than inferring cache behavior.
- Bound the live context. Stateful does not mean infinite; compaction and clean recovery remain part of the architecture.
The broader implication
Stateful inference changes more than latency. It changes the contract between an agent and its model. Instead of repeatedly describing a world to a fresh inference call, the agent can maintain a living computational state and deliver only what changed.
That is especially interesting for local assistants, scientific copilots, and event-driven agents that listen, use tools, and stay active over long sessions. The closer an agent gets to continuous work, the less sensible full transcript replay becomes.
Open questions
How should state move between devices or workers? When is a semantic compaction worth the cost of rebuilding KV? How can tool-result branches be reconciled safely? And how does resident state compare with prefix caches that must be restored across a distributed fleet?
Those are the next experiments. Stateful inference already looks less like a runtime trick and more like an architectural primitive for agents that are meant to remain present.