How Vocabulary Specialization Cut SLM Inference Cost by 77% in a Production Grader

How Vocabulary Specialization Cut SLM Inference Cost by 77% in a Production Grader

Models specialized for a domain or task by fine-tuning general open-source systems, such as Qwen and Gemma, retain vocabularies designed for many domains and general applications. In a production essay grader, that vocabulary split frequently generated text into more tokens than necessary.

A method called VocabX specialized the vocabulary around the cost of each token rather than its frequency. Using the same base model, data, and training recipe, it reduced measured inference cost by 77% without changing the base model.

The Production Use Case

The system is an automated grader for Portuguese essays built on Qwen3-4B. It receives an essay, its prompt, and supporting texts, then returns a JSON assessment containing a grade and comment for each competency.

The model has 4 billion parameters and was already trained for the task. Its tokenizer, however, remained the original Qwen3-4B tokenizer, designed for many languages and text types. In this application, it handled only Portuguese and one response format.

This created two costs:

• Generated text used more tokens than necessary.
• At every generated token, the model scored a vocabulary of 151,669 entries, including many that the grader rarely used.

VocabX rebuilt the vocabulary while keeping the base model, data, and training recipe unchanged. The reported results come from the current version of the paper, which was still under review and could change before publication.

A Vocabulary Built for General Use

Language models process text as tokens, and the tokenizer determines how text is divided. General-purpose tokenizers are trained on large collections covering many languages and domains, including source code, contracts, and Chinese. That design is appropriate for a general-purpose model.

For a model serving one domain, it can create two forms of waste.

Fragmentation

The tokenizer may split into many pieces the text that the application repeats most often. In the grader, every response begins with the same JSON envelope, competency names, and evaluation phrases. In contracts, the repeated material may be boilerplate clauses. In code, it may be identifiers.

These sequences require several tokens where one could suffice, and the model pays the cost on every request.

Excess Vocabulary

The vocabulary also contains tens of thousands of entries that the application rarely or never uses. A model tuned to grade American school biology exams, for example, will almost never generate Chinese, yet it still computes a score for every Chinese-language token at each generation step.

The grader's base tokenizer contained 151,669 entries. VocabX reduced that number to 64,767.

Why Generated Tokens Cost More

Counting tokens is not the same as measuring inference cost. Inference has two main phases: prefill and decode.

During prefill, the model processes the complete input in one pass. Each weight read from GPU memory can be reused for multiple input tokens, so computation is generally the limiting factor.

During decode, the model generates one token at a time. Each token requires another pass through the network. Weight reads are reused only a few times, and the GPU spends much of its time waiting for data from memory. The limiting factor therefore becomes memory bandwidth.

This effect is particularly significant for small models served with moderate batch sizes. Batching distributes weight reads across requests, but does not eliminate the problem. Each decode step also reads the KV cache, which stores the processed context and grows as the sequence continues.

In the measured grader configuration, generating one token cost 7.4 times as much as reading one token. The ratio varies with hardware, task, batch size, context length, and other factors, and can be higher. In general, generated tokens cost more than input tokens.

The model's vocabulary adds another cost. Its final layer, the LM head, converts the internal state into a score for every possible token. It contains one parameter row for each vocabulary entry. In a model the size of Qwen3-4B, this matrix accounts for almost 10% of the parameters, which is significant for a small model.

The input layer looks up only the row associated with each token being read. The LM head, by contrast, reads and multiplies the full matrix for every generated token. Because decode is memory-bound, every additional vocabulary row creates more data to move at each step.

As a result, adding tokens can reduce the token count while increasing the inference bill. VocabX did not include this term directly in its selection criterion, but it controlled vocabulary size in two ways:

1. A sub-word merge replaced the base vocabulary with a smaller, domain-specific vocabulary.
2. Macro tokens were added under a fixed budget.

For the grader, the output layer fell from approximately 152,000 rows to approximately 65,000.

Limitations of Existing Approaches

Vocabulary adaptation is not new. Methods such as fast vocabulary transfer (FVT), multi-word tokenization (MWT), and Liquid AI's in-place tokenizer expansion, also called in-place BPE, can reduce fragmentation by continuing BPE training from an original vocabulary.

Testing these approaches on the same problem revealed two limitations.

Frequency Is Not the Same as Cost

Existing methods generally choose tokens according to frequency rather than serving cost. An expression that appears often in the input may therefore be considered more valuable than one that appears less frequently in the response, even when the response token is more expensive to generate.

Separate Layers Are Optimized

The methods typically adjust either how words are divided into sub-word pieces or how complete phrases are merged into a single token. They do not optimize both layers under one criterion.

The grader's largest inefficiency came from long, repeated spans such as the JSON envelope. A sub-word approach can reach these spans only by merging their components one at a time.

How VocabX Works

VocabX addresses both limitations and adds a method for handling candidates that are missed during sampling.

Cost-Weighted Selection

Instead of asking how many tokens a candidate saves, VocabX estimates how much cost and time it saves. Each eliminated token is weighted according to the field where it occurs: the system prompt, user message, or model response.

In the measured grader, a token saved in the response was worth 7.4 times a token saved in the input. The vocabulary was selected accordingly.

Sub-Word and Multi-Word Optimization

VocabX works at two levels:

• At the sub-word level, it rebuilds how domain-specific words are divided into pieces.
• At the multi-word level, it identifies repeated spans, from short expressions to complete lines and JSON envelopes, and converts each into a single macro token.

Both layers use the same cost criterion, with a separate budget for each.

Five automatic miners search the corpus for different kinds of repetition:

• Long words
• Complete lines
• Expressions
• Complete sentences
• Punctuation-based structures, including JSON and tags

Across eight corpora, the punctuation and structure miner contributed the most on average. On code, it accounted for 17.5 points of reduction by itself. Removing all five miners reduced token reduction on the grader by 55.6 points.

Estimating Unobserved Candidates

Cost-based selection requires estimating how much each candidate saves after previously selected tokens are taken into account. Measuring this exactly would require reprocessing the entire corpus after every choice, potentially tens of thousands of times.

A scalable alternative is to evaluate candidates on a document sample. However, candidates absent from the sample receive no measured gain. An expression appearing 37 times in 50,000 documents appears in a 400-document sample only about 26% of the time. In a contracts corpus, only 16.6% of candidates appeared in a 400-document sample.

VocabX separates a candidate's gain into two parts:

1. The total saving across the corpus, which can be counted exactly.
2. The portion not already covered by selected tokens, which must be estimated.

For candidates missing from the sample, VocabX uses a statistical estimate based on candidates that the sample does contain instead of assigning zero gain. A candidate that is rare in the sample but frequent in the full corpus can therefore remain under consideration.

Using the same sample and budget, this change increased token reduction on a contracts corpus from 44.7% to 52.8%.

Results on the Production Grader

Seven grader versions were trained with the same base model, data, recipe, and seed. The tokenizer was the only difference between versions.

Each version processed the same 500 requests on the same test machine, equipped with an RTX 5090 GPU and the vLLM inference server. Cost was calculated from measured runtime.

The average model response fell from 350 tokens to 101 tokens. Measured inference cost declined by 77.0%. Quality, measured by a bench score combining grading accuracy with the accuracy of screening out off-topic or rule-breaking essays, increased from 0.703 to 0.810.

The result required an adjustment to the initial vocabulary design. The first version fused each competency's grade with the surrounding JSON into one macro token. Although this was the most compact option, quality declined to a bench score of 0.689, below the production grader's 0.703.

When the grade was included in the macro, every combination of grade and structure became a separate token variant. The model selected both in one step. Giving the grade its own token added two tokens per response, but increased the bench score to 0.810. It also produced a larger measured cost reduction, 77.0%, compared with 73.3% for the fused version.

The result shows that compression can remove information needed for the model's central decision. The objective is not the smallest possible vocabulary, but the least expensive vocabulary that preserves important decisions.

Results Across Other Domains and Models

VocabX was also compared with the three reference methods across five public domains:

• Contracts
• Medical papers
• Scientific papers
• News
• Python code

The comparison covered three model families: Qwen3, Llama-3.1, and Gemma-3. All methods used the same mining and test documents, along with the same vocabulary ceiling.

VocabX produced more compression than the strongest reference method in all 15 combinations, with gains ranging from 1.2 to 9.3 points.

The pruned version of VocabX also led in estimated inference cost across all 15 combinations. Its average margins were:

• 10.2 points on Qwen3
• 7.5 points on Llama-3.1
• 13.6 points on Gemma-3

On contracts, the pruned version achieved 62.6% token reduction with 113,426 vocabulary entries. The best in-place BPE configuration achieved 61.8% reduction while using 207,803 entries.

Vocabulary as an Optimization Layer

VocabX does not change the model's intelligence. It reduces the amount of text the model must generate piece by piece and the number of vocabulary entries it must score during generation.

In the Portuguese essay grader, this reduced measured inference cost by 77% while keeping the same base model and training recipe. The results suggest that model specialization can begin not only with data and weights, but also with the vocabulary layer.

The VocabX paper, including its full method, experiment tables, and discarded interventions, was still in preparation in 2026.

Sources

• Dharma-AI, "VocabX: Cost-Weighted Vocabulary Adaptation for Domain-Specific Language Models," in preparation (2026).
• Leonidas Gee et al., "Fast vocabulary transfer for language model compression," EMNLP Industry Track (2022).
• Leonidas Gee et al., "Multi-word tokenization for sequence compression," EMNLP Industry Track (2023).
• Jimmy T.H. Smith et al., "In-Place Tokenizer Expansion for Pre-trained LLMs," arXiv:2607.15232 (2026).