NeoMME: An Efficient Multimodal-Native and Multilingual Encoder

NeoMME: An Efficient Multimodal-Native and Multilingual Encoder

Published September 3, 2026

Overview

NeoMME is a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models, NeoMME does not use a separately pretrained vision tower or a causal language model. A single bidirectional Transformer processes text tokens and raw image patches, and the complete model is trained from scratch with a masked discrete-diffusion objective.

A fine-tuned version, NeoMME-Retriever, applies the page-image approach used by ColPali to visual document retrieval. It produces dense and late-interaction embeddings in one forward pass. Both model sizes are on the ViDoRe v3 Pareto frontier for nDCG@10 and model size.

With 2048×2048 images on an NVIDIA L40S GPU, NeoMME-Retriever-260M encodes about 51 pages per second, nearly twice ColModernVBERT's 26 pages per second. Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from approximately 1.5 MB to 6 kB per page, a 255× reduction, while retaining more than 95% of baseline nDCG@10.

NeoMME is available through Hugging Face Transformers, and all model checkpoints are released under the Apache 2.0 license.

Why another multimodal encoder?

Many visual document retrievers are adapted from pretrained generative visual language models. In that design, a separately pretrained vision encoder creates visual features, a projector maps them into the language model's input space, and a causal decoder processes the combined image and text representations.

Retrieval, classification, and token labeling do not generate text autoregressively. They therefore do not require a causal decoder or the additional parameters and computation associated with that architecture.

ModernBERT introduced architectural and training improvements for efficient bidirectional encoders. ModernVBERT applied a bidirectional ModernBERT-style text encoder to visual document retrieval while retaining a separately pretrained SigLIP2 vision tower. NeoMME extends this direction with a multimodal encoder trained without a separate pretrained vision tower, text encoder, or text decoder.

Pronounced "nee-oh-me" (IPA /ˈniː.oʊ.mi/), NeoMME is a multilingual, multimodal foundation encoder that creates vector representations for text, images, or both. Image patches and text tokens pass through the same bidirectional Transformer.

This shared computational path can simplify pretraining, fine-tuning, parallelization, and serving across modalities.

NeoMME encoder architecture

NeoMME is available in two sizes: 260M and 800M. Both use the same general architecture.

One Transformer for images and text

• Native multimodal inputs: Text uses factorized token embeddings. Images are divided into non-overlapping 32×32 patches and projected through a small MLP. Both representations enter the same Transformer encoder.
• Dynamic image resolution: Images retain their aspect ratio and size. High-resolution, information-dense document pages can therefore use more tokens than smaller images with less content.
• Long bidirectional context: Both models support 16,384 tokens, enough for up to two standard 3840×2160 4K UHD images. Most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention.
• Modern encoder components: The architecture includes grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs, among other improvements.
• Multilingual text: A BPE tokenizer with a 131k-token vocabulary was trained from scratch on multilingual text, code, mathematics, and machine-produced image transcripts.

Learning from images through masked text

NeoMME is pretrained from scratch as a discrete masked-diffusion text denoiser. For each text-only example, a corruption rate is sampled uniformly between 0 and 1. Each eligible text token is then independently masked at that rate.

For multimodal examples, corruption rates range from 0.3 to 1. Image patches remain visible while the model reconstructs masked text. With light masking, surrounding text may be sufficient to infer a missing word. For example, "cat" is a plausible completion for "The [MASK] sat on the mat" without using an image. Higher masking rates reduce this language-only shortcut and encourage the model to learn image-grounded descriptions.

Pretraining combines multilingual text, code, mathematics, natural images, and document images. Each model processes approximately 524 billion packed input tokens, including 290 billion tokens from text-only examples. This is smaller than ModernBERT's 2 trillion training-token budget, so the NorMuon optimizer was selected to improve training data efficiency.

NeoMME-Retriever

NeoMME was fine-tuned for visual document retrieval using the page-image methodology introduced by ColPali. Instead of retrieving extracted text chunks, NeoMME-Retriever ranks document page screenshots and avoids the OCR preprocessing normally required for PDF text extraction.

Treating pages as images preserves layout, charts, tables, font type, font size, and other visual information that OCR may not capture fully.

Dual heads for dense and late-interaction retrieval

NeoMME-Retriever adds two jointly trained retrieval heads to the NeoMME backbone:

• Dense head: Averages the backbone's hidden states through mean pooling and produces a normalized vector. Dense embeddings are compact and work naturally with approximate nearest-neighbor (ANN) indexes.
• Late-interaction head: Projects each text token or image patch into a 128-dimensional normalized vector. This finer-grained representation preserves local matches between query tokens and image regions.

A single forward pass returns both representations. Late-interaction embeddings are generally more expressive and can be used with open-source libraries such as NextPlaid. For very large corpora, a dense embedding can first retrieve a small candidate set through an ANN index, after which late interaction can rerank those candidates.

Retrieval performance at compact model sizes

On ViDoRe v3, NeoMME-Retriever-260M reaches an nDCG@10 score of 0.523, the highest score among evaluated models strictly below 800M parameters. It is within 0.002 nDCG@10 of ColQwen2.5 while using approximately 14 times fewer parameters.

NeoMME-Retriever-800M reaches 0.556, within 0.009 nDCG@10 of the similarly sized Vultron Retriever Flash (0.8B). Both NeoMME-Retriever models are on the model-size Pareto frontier.

ViDoRe v1 and v2 use nDCG@5. On both benchmarks, NeoMME-Retriever-260M outperforms ColModernVBERT and the twice-larger ColSmol-500M. NeoMME-Retriever-800M outperforms ColPali v1.3 while using 3.6 times fewer parameters.

Scores marked in the original evaluations were obtained from MTEB or through the authors' own evaluations. The table preserves the reported values.

Making high-resolution late-interaction retrieval practical

Late-interaction storage grows with the number of vectors in an output embedding. Higher-resolution images contain more patches and therefore produce larger embeddings. A 2048×2048 square page produces approximately 4,200 vectors with NeoMME-Retriever, requiring about 2.1 MB in float32. Across ViDoRe v3, measured storage averages approximately 1.5 MB per document.

Two compression methods reduce this footprint:

1. Hierarchical token pooling: Similar document vectors within a multi-vector embedding are clustered and replaced by their cluster mean, reducing the number of stored vectors per page.
2. Asymmetric quantization: Document embeddings are quantized to int8 or binary. Query embeddings are generated when needed rather than stored, so they can remain at higher precision.

On ViDoRe v3, pooling factor 10 with int8 queries and documents reduces storage from about 1.5 MB to 39 kB per page, a 39× reduction, while retaining more than 99% of baseline nDCG@10.

A more aggressive configuration uses pooling factor 8, int8 queries, and binary documents. It requires approximately 6 kB per page, 255 times smaller than the baseline, while retaining more than 95% of the original retrieval quality.

Inference speed for corpus indexing

Before a corpus can be searched, a retriever converts its documents into embeddings that are stored in a vector database such as Qdrant, Weaviate, or Milvus. Faster encoding reduces the time and GPU compute required to build or update an index.

At a matched 2048×2048 input size on one NVIDIA L40S, NeoMME-Retriever-260M encodes about 51 pages per second, compared with 26 pages per second for ColModernVBERT. Both NeoMME-Retriever-260M and NeoMME-Retriever-800M are also faster than the other compared models on smaller input images.

Using NeoMME-Retriever

NeoMME-Retriever-260M and NeoMME-Retriever-800M return dense and multi-vector embeddings together. The following example scores two text queries against two document-page images using MeanMaxSim late interaction and dense cosine similarity.

pip install -U accelerate "transformers @ git+https://github.com/huggingface/transformers.git@main" "sentence-transformers>=6.0.0"

from typing import Any, Literal

import requests
import torch
from PIL import Image
from sentence_transformers.util import cos_sim, mean_maxsim
from transformers import BatchFeature, NeoMMEForRetrieval, NeoMMEProcessor

def encode(
    messages: list[list[dict[str, Any]]],
    task: Literal["query", "document"],
) -> BatchFeature:
    return processor.apply_chat_template(
        messages,
        task=task,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        processor_kwargs={"padding": "longest"},
    )

model_name = "Hcompany/NeoMME-260M-Retriever"
processor = NeoMMEProcessor.from_pretrained(model_name)
model = NeoMMEForRetrieval.from_pretrained(model_name, device_map="auto")

image_urls = [
    "https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/shift_kazakhstan.jpg?raw=true",
    "https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/energy_electricity_generation.jpg?raw=true",
]
documents = [Image.open(requests.get(url, stream=True).raw) for url in image_urls]

queries = [
    "Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?",
    "Which hour of the day had the highest overall electricity generation in 2019?",
]

document_messages = [
    [{"role": "user", "content": [{"type": "image", "image": document}]}]
    for document in documents
]
query_messages = [[{"role": "user", "content": query}] for query in queries]

inputs_documents = encode(document_messages, "document").to(model.device)
inputs_text = encode(query_messages, "query").to(model.device)

with torch.inference_mode():
    document_outputs = model(**inputs_documents)
    query_outputs = model(**inputs_text)

late_scores = mean_maxsim(
    query_outputs.embeddings,
    document_outputs.embeddings,
    a_mask=inputs_text["attention_mask"],
    b_mask=inputs_documents["attention_mask"],
)
dense_scores = cos_sim(
    query_outputs.dense_embeddings,
    document_outputs.dense_embeddings,
)

print(late_scores, dense_scores)

The expected ranking is that the first query scores higher against the first document and the second query scores higher against the second document.

Fine-tuning with Sentence Transformers

Separate dense and late-interaction checkpoints are available for fine-tuning with Sentence Transformers v6.

Following the pattern used for text encoders such as ModernBERT, Sentence Transformers loads the backbone through NeoMMEModel rather than the dual-head NeoMMEForRetrieval class. Sentence Transformers currently supports one retrieval head per model, so the dense and late-interaction checkpoints can be fine-tuned independently. Training both heads together requires NeoMMEForRetrieval with a custom Trainer.

From retrieval to visual RAG

Visual document retrieval can serve as the first stage of a visual retrieval-augmented generation (RAG) system. Instead of retrieving extracted text chunks, visual RAG retrieves original page images and sends them to a visual language model. This allows the model to use tables, plots, diagrams, and page layout that text extraction may flatten or omit.

A visual RAG pipeline typically has three stages:

1. Indexing: Convert each PDF page into an image, generate an embedding with a retrieval model, and store the embeddings in a vector store.
2. Retrieval: Generate an embedding for the user's query with the same model and retrieve the top-k relevant pages.
3. Generation: Append the retrieved images after the query, for example {query}{img_1}{img_2}...{img_k}, and send the combined input to a VLM to generate an answer.

Conclusion

NeoMME uses one long-context bidirectional Transformer in place of separate pretrained image and text encoders. It is trained from scratch on multilingual text tokens and raw 32×32 image patches.

NeoMME-Retriever is fine-tuned for visual document retrieval and produces dense and late-interaction representations in one forward pass. The 260M model outperforms all evaluated models strictly below 800M parameters. At a matched 2048×2048 input size, it encodes pages at approximately twice the throughput of ColModernVBERT.

Hierarchical token pooling and asymmetric quantization reduce high-resolution late-interaction embeddings from roughly 1.5 MB to 6 kB per page, a 255× reduction, while retaining more than 95% of baseline nDCG@10.

The NeoMME model checkpoints and a Hugging Face Transformers implementation are released for building multimodal and multilingual representation models.

Acknowledgements

NeoMME was developed by Aurélien Lac and Tony Wu with support from H Company, including the compute used to train the models.

Citation

@misc{lac2026neommesingletowermultimodalnativemultilingual,
      title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},
      author={Aurélien Lac and Tony Wu},
      year={2026},
      eprint={2609.01657},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2609.01657},
}