Tokenizers v1: Encoding, Decoding, and Scaling Measured

Published September 21, 2026

Tokenization has not traditionally been a bottleneck in machine learning workflows. Compared with model execution, converting text into tokens usually requires relatively little computation. As models become faster and workloads grow, however, tokenization can begin to limit overall throughput.

Large-scale training, highly concurrent serving, and repeated processing of long inputs can place enough pressure on the tokenizer that the model waits for the CPU to provide data. Tokenizers v1 focuses on reducing that delay and scaling with the surrounding workflow.

This article summarizes the release candidate's performance compared with tokenizers v0.23, including encoding, decoding, multithreading, scaling, latency, memory use, and crate size. Many workloads show improvements of several times, with some reaching tens of times faster.

The benchmarks use the tokbench repository. The repository includes a command for rerunning the measurements on other hardware.

What Tokenizers v1 Is

Tokenizers v1 produces the same token IDs as v0.23. The project preserves the existing output, API, vocabulary, and merge ranks while improving the implementation. It remains general across tokenizer families instead of specializing in byte pair encoding (BPE), and it loads everything supported by v0.23.

A tokenizer converts text into the integer sequence consumed by a model. The tokenizers pipeline has four stages:

Normalization applies transformations such as lowercasing or Unicode normalization.
Pre-tokenization divides text into smaller pieces called pre-tokens.
Model processing converts each pre-token into tokens and maps them to vocabulary IDs.
Post-processing adds special tokens required by the model.

Most of the performance work described here affects the model stage. Eight of the ten measured model families use BPE. BPE begins with the bytes of a pre-token and repeatedly joins the highest-ranked adjacent pair until no ranked pair remains. The rankings are learned during tokenizer training and shipped with the tokenizer, so the same input produces the same IDs. Merges never cross pre-token boundaries.

The other two measured families use WordPiece and Unigram, the two other model types supported by the library.

Main Implementation Changes

Every pipeline stage received attention. The changes with the largest performance impact include:

Change	What it does
Workspace split	Divides one crate into a workspace. `tk-encode` is required at runtime, while `tk-serialize`, `tk-convert`, and `tk-train` are linked only when needed.
Allocation-free model processing	Keeps the merge working set in caller-owned scratch memory, allowing the merge loop to avoid the allocator.
Bitcannon	Represents the split pattern with Boolean operations over bitstreams and uses SIMD instructions to identify boundaries instead of a general-purpose regular-expression engine.
Merge-loop rewrite	Stores mergeable pieces in an intrusive doubly linked list inside one preallocated buffer, so a merge updates indices instead of moving data.
Word cache	Keeps a thread-local mapping from pre-token bytes to completed IDs, allowing repeated words to bypass the merge process.
Native parallelism	Allows one shared tokenizer to encode from multiple threads. Each thread receives scratch memory and cache resources from its own sub-pool, avoiding contention on a single lock.

Bitstreams Instead of a Regular Expression

BPE models use a regular expression to divide input into smaller pre-tokens. Merges occur only within a pre-token, so this split determines the boundaries visible to the rest of the pipeline.

The regular expression is a fixed model parameter. It is stored with the tokenizer and does not change during encoding, which means a general-purpose regular-expression engine does not need to interpret it for every input. A tokenizer can instead use a specialized splitting function for the pattern associated with that model.

The handwritten function can use SIMD, or single instruction, multiple data, instructions available on modern CPUs. SIMD applies one operation to many bytes at once and is well suited to UTF-8 text. Bitcannon treats the input bytes as parallel bit streams and identifies boundaries through Boolean operations across entire registers rather than scanning one character at a time. One register operation processes 64 bytes.

The approach is related to techniques used by Parabix for text processing and simdjson for JSON parsing.

This optimization depends on recognizing the model's pattern. A small number of grammars cover many byte-level BPE models. If a tokenizer uses an unsupported pattern, it continues to use the regular-expression path and does not receive this optimization. This is one reason performance gains differ between model families.

The Word Cache

Natural language contains repeated words. Since BPE produces the same IDs for a given pre-token, tokenizers v1 can store the result after processing it once. A thread-local cache maps pre-token bytes to token IDs, allowing later occurrences to skip merging.

As an input grows, the number of unique words may grow more slowly than the total number of words. Repeated words therefore account for a larger share of the workload, although new words still cause cache misses.

Caching is most effective when inputs contain many repeated pre-tokens. Inputs with little repetition can incur lookup costs without receiving many cache hits.

The Merge Loop

The BPE merge loop repeatedly finds the highest-priority adjacent pair in each pre-token and merges it. The previous implementation allocated memory for every call and created a new priority queue for every pre-token.

Version 1 reuses scratch memory supplied by the caller, eliminating those repeated allocations. Symbols are stored in a flat array, while links between adjacent symbols are represented by positions in that array. This makes updates during merging less expensive. The implementation also processes batches of pre-tokens in one model call.

Each candidate pair is packed into a 64-bit value, with the merge rank stored in the high bits. Comparing candidates then requires only integer comparisons. The value representing no available merge is the largest possible value, allowing the loop to select its next merge without a branch.

Benchmark Method

Small changes in benchmark design can produce substantial differences in measured tokenizer performance. The comparison used the following rules:

Rule	Purpose
One timing loop	Every engine runs the same loop, without engine-specific fast paths.
Loading excluded	Vocabulary loading is timed separately and never included in encoding time.
Output hash verified	FNV-1a over the output IDs must exactly match the baseline.
Common cells only	Medians include only measurements that every engine ran and verified.
Complete sweep per process	Each repeat starts in a new process, which retains every measured cell.
Physical-core pinning	Workers are pinned to eight distinct physical cores rather than sibling SMT threads.
Independent jobs	Separate jobs measure variation between hosts.

Repeatedly encoding the same document can be faster than encoding a stream of distinct documents on the same build. The first workload represents a document already fully represented in the cache. The second measures new input while allowing previously seen pre-tokens to remain cached.

Both workloads are sometimes called warm, although they represent different conditions. The headline results use distinct documents, and the complete corpus is too large to fit in the cache. Tokenizer benchmarks should identify which workload they measure because this choice can materially affect the result.

Results

Across the ten model families covered by the v1 encoding path, tokenizers v1 encodes text 3 to 30 times faster than v0.23 in single-threaded tests on an Apple M4 Max. The lower end of that range is t5-base, while the upper end is gpt2.

Across eight workers, the implementation reaches 76% of linear scaling. The output remains exactly the same as the released library, including the token IDs.

The overall improvement comes from several changes working together:

A handwritten splitter replaces regular-expression processing for supported patterns.
The word cache avoids repeating the merge process for previously seen pre-tokens.
The merge loop reuses caller-provided memory instead of allocating repeatedly.
Batched model calls process multiple pre-token spans together.

Each change reduces work at a different point in the pipeline.

Installation and Usage

A release candidate for v1 is available on crates.io. The standard installation is:

cargo add tokenizers --pre

Training is enabled by default and brings in a C++ dependency. Applications that only need encoding can disable the default features and enable HTTP support:

cargo add tokenizers --pre --no-default-features --features http

The encoding API is unchanged. The following example loads a tokenizer and prints its IDs and tokens:

use tokenizers::tokenizer::{Result, Tokenizer};

fn main() -> Result<()> {
    let tokenizer = Tokenizer::from_pretrained(
        "deepseek-ai/DeepSeek-V4-Flash",
        None,
    )?;

    let encoding = tokenizer.encode(
        "The tokenizer is no longer the bottleneck.",
        false,
    )?;

    println!("{:?}", encoding.get_ids());
    // [671, 17840, 9160, 344, 1119, 5827, 270, 111127, 16]

    println!("{:?}", encoding.get_tokens());
    // ["The", "Ġtoken", "izer", "Ġis", "Ġno", "Ġlonger", "Ġthe", "Ġbottleneck", "."]

    Ok(())
}

For batches, encode_batch is the API used to scale encoding across cores:

let encodings = tokenizer.encode_batch(documents, false)?;

The benchmark figures measure the Rust crate directly. The Python bindings use the same implementation through bindings/python, but calls through those bindings add per-call overhead that is not included in these measurements.

Progress Toward v1

The release-candidate work described in the benchmarks has been implemented in the Rust pre-release on crates.io.

Implemented for the Release Candidate

Workspace split: Divides the original crate into tk-encode, tk-serialize, tk-convert, and tk-train, allowing applications to link only the components they use.
Bitcannon: Replaces regular-expression splitting on the encoding path with bitstream operations for GPT-2, cl100k, o200k, Tekken, and DeepSeek. It replaced the finite-state machines used in earlier versions.
WordCache: Reuses token IDs for previously processed pre-tokens.
Faster lookup and merging structures: Adds FlatCache, MPHF RankStore, incremental merging, and BucketVocabStore.
Reusable model memory: Moves temporary model state into scratch buffers so tokenization does not allocate on every call.
Pipeline post-processing: Exposes post-processing as the STAGE_POST pipeline stage.
Batched model calls: Processes multiple pre-token spans in one call.
Faster decoding: Writes decoded bytes directly into a reusable buffer, avoids intermediate strings and copies, accelerates token lookup, supports buffered streaming, and decodes batches in parallel.
role_to_token support.
Node.js bindings.

Planned for 1.0.0

One encoding implementation: Uses tk-encode during training validation so training and inference cannot produce different tokenization results.
Optional offsets and masks: Computes this metadata only when requested, keeping it out of the token-ID-only path.
Normalizer rework.
Bitnorm support: Builds on atomnorm.
Precompiled SentencePiece support.
Simpler Python bindings: Reduces locking, wrapper types, and handwritten dispatch code while preserving subclassing, serialization, custom decoders, mutation behavior, and support for free-threaded CPython.
Inference-only C and C++ bindings: Targets ExecuTorch and llama.cpp, with possible JVM, Swift, and Go bindings afterward.

After 1.0.0

The project may explore tok-devices, which would target GPU encoding and batch decoding while keeping text and token IDs on the device. The decoder would upload the vocabulary once, calculate output positions in parallel, and gather the corresponding bytes on the GPU. This would be an optional component intended for large batches and would require further prototyping and measurement.

Future work also includes moving additional model families onto the new merge loop before 1.0.0, followed by integrating the improvements into the transformers library and other components that depend on tokenizers once the release candidates stabilize.

Tokenizers v1: Encoding, Decoding, and Scaling Measured