Sub-5ms inference on CPU — STAN's deployment story

The hardest thing about deploying STAN-1-Mini was making it invisible.

ContextAPI's value prop is "your bill drops by 50%, your latency stays the same." If we add 50ms of inference overhead to do prompt analysis, we eat half the latency budget before the LLM call. So the deployment requirement was sub-5ms p99 — and that had to hold under bursty production load on commodity CPUs.

We hit it. Here's the short version of how.

The bottleneck nobody talks about

When you profile a small ML model in production, the model inference is not the bottleneck. The tokenizer is.

For STAN-1-Mini, the model itself takes about 1.2ms on a single CPU core. Tokenization, when we started, was taking 8ms. The 12M-parameter neural network was doing less work than the regex-based pre-processing.

We rewrote the tokenizer in Rust, exposed via PyO3. New cost: 0.4ms. That single change moved us from p99 of 14ms to p99 of 6ms.

ONNX runtime

We trained the model in PyTorch. We didn't ship it in PyTorch.

ONNX runtime with the CPU execution provider is meaningfully faster than PyTorch for small inference workloads — primarily because it can fuse operators that PyTorch leaves separate. For our model architecture, the speedup was ~2.5×.

The conversion process is mostly painless if you stay inside the ops ONNX supports. We had to rewrite one custom layer that used a fancy indexing pattern PyTorch handles natively. Worth it.

Quantization

Post-training INT8 quantization with per-channel calibration. We quantised everything except the final classification heads (which we kept in FP32 because they're tiny and the precision matters for the policy outputs).

Quantization dropped model size from 48MB to 13MB and inference time by another 30%. We saw no measurable degradation in downstream policy quality on our eval set.

Threading model

The inference path runs on a dedicated thread pool, separate from the main request thread pool. This matters because ContextAPI's main thread is doing a lot of work — parsing PromptIR blocks, running the rule-based compression, talking to the LLM. We don't want STAN inference contending with that.

Each inference task takes a single CPU core and runs to completion. We deliberately do not batch — batching would add latency on the per-request critical path, and our QPS is high enough that we get throughput from concurrency, not batching.

Cache pre-warming

The first STAN-1-Mini inference on a fresh worker takes ~80ms because the ONNX runtime is JIT-compiling kernels and pulling weights from disk. We pre-warm by running a dummy inference at worker startup, before the worker accepts traffic.

That cuts cold-start to <2ms. Workers that pass health checks have already done their warm-up.

Failover

What happens if the ONNX runtime crashes, the worker hangs, or weights fail to load? The pipeline falls back to a calibrated heuristic. The fallback is the rule-based system we used before STAN existed. It scores worse but doesn't fail.

We monitor fallback rate as a SLI. Current rate is <0.01%.

Numbers

Final p99 inference latency: 4.2ms on a single core of a generic x86 worker. Cold-start: <2ms. Memory footprint: 18MB per worker (quantised weights + ONNX runtime overhead).

The deployment story is rarely the headline of an ML system — but it's where these systems actually win or lose. STAN-1-Mini works in production because the deployment was treated as a research problem of its own.