STAN-1-Mini: Training a 12M-parameter RL policy to compress prompts

Most prompt-compression systems we benchmarked at the start of 2024 were heuristic stacks dressed up as research. Strip stopwords. Remove filler phrases. Truncate context. Maybe run an LLM-as-compressor pass. The results felt impressive on cherry-picked examples and collapsed everywhere else.

We wanted something that actually adapted to the prompt in front of it — light touch on lean prompts, aggressive on bloated ones — without dragging a 70B model into the inference path.

Why a 12M-parameter model

The unit of work we cared about wasn't language generation. It was a small set of structural decisions: how messy is this prompt, how much can we afford to compress, which tier of compressor should handle it. That's a classification problem dressed up as policy learning. You don't need a frontier model to make those calls.

We sized STAN-1-Mini against three constraints:

• ›Sub-5ms inference on CPU. Adding even 20ms of overhead before the LLM call kills the business case for compression.
• ›Fits in a single CPU cache line of activations. Means it runs on a worker thread without contending for the main inference path.
• ›Trainable on a single H100 in under 12 hours. Iteration speed matters more than parameter count for this class of problem.

12M parameters threaded those three needles. Going smaller hurt the messiness score's accuracy on adversarial prompts. Going larger added latency without lifting the F1.

The reward function

Every paper on RL-for-compression we read used proxy rewards: ROUGE against a target, BLEU against a reference, or task-accuracy on a downstream benchmark. We tried all three. They all fail in production.

The reward we ended up shipping has four terms:

• ›Compression ratio. Tokens removed divided by input tokens. We want this high, but only conditionally.
• ›Intent preservation. A judge model compares the optimised prompt to the original and scores whether a downstream model would answer the same question. Cosine similarity on embedding-space wasn't enough — we needed a tasked judge.
• ›Latency penalty. Compression that takes longer than the savings it produces is negative-value. The reward subtracts a wall-clock cost.
• ›Calibration term. If the policy emits a compression target of 60% but the prompt is already clean, that's a bad call even if compression succeeds. We penalise mismatched signals.

The calibration term was the unlock. Without it, the policy learns to be aggressive everywhere because aggressive compression scores well on average. With it, the policy learns when not to compress — which is most of the time.

What auto mode actually does

When a request arrives at ContextAPI, STAN-1-Mini reads three things: the raw token count, the typed PromptIR block structure, and a small set of structural features (entropy, repetition rate, named-entity density). It outputs three numbers:

• ›A compression target between 10% and 90%.
• ›A messiness score between 0 and 1.
• ›A priority signal that routes to cosavu-small, cosavu-medium, or cosavu-large.

The rest of the pipeline is deterministic. Block rewriting, PII scrubbing, governance — those are all rule-based passes that respect the targets STAN sets.

Results

On our internal benchmark (2,400 production prompts sampled across customer tenants), STAN-1-Mini lifts average compression from 31% (rule-based baseline) to 48% with no measurable drop in downstream task accuracy. p99 inference latency is 4.2ms on a single CPU core.

The bigger result, though, is that the variance dropped by a factor of 3. The rule-based baseline either over-compressed (losing intent) or under-compressed (saving nothing). STAN-1-Mini stays in the productive band on 96% of inputs.

What's next

STAN-1-Mini is shipping in ContextAPI now — every optimize call is routed through it. The next version (STAN-1, full-size) is in training and adds streaming compression for very long contexts. We'll write that one up when it lands.