Build Specialized Multimodal Models with LFM3

Build Specialized Multimodal Models with LFM3

Published October 27, 2026

LFM3 is the latest generation of Liquid Foundation Models (LFMs), designed as a starting point for specialized intelligence. The family brings text, vision, audio, omni, and decision models together on a shared backbone, with all models in the 2B-parameter range. Developers can select a model for a specific modality, fine-tune it on their own data using a single GPU, and deploy it where the application runs.

LFM3 introduces four main changes compared with the LFM2 family:

• More capability per parameter: A new tri-hybrid architecture and training recipe allow LFM3-1.7B to outperform LFM2.5-2.6B with 35% fewer parameters.
• One backbone for every modality: Six 2B-class models for text, vision, audio, omni, and decision tasks are trained from LFM3-1.7B-Base. The family includes the first LFM omni model and a vision model with video input.
• Designed for edge devices: The text model requires about 1 GB at 4-bit precision. New in-house vision and audio encoders improve processing speed, while the text model's 256K context window supports on-device agents.
• Built for specialization: Every model can be fine-tuned on a single GPU and adapted to a specific dataset, terminology, or deployment environment.

A more expressive backbone

LFM2 demonstrated that combining short convolutions with attention can provide fast on-device inference without sacrificing quality. For LFM3, hardware-in-the-loop architecture search was used to redesign that approach for agentic workloads, which require greater capability per parameter and efficient inference over long contexts.

The search now covers 14 device and backend combinations, including phones, a Raspberry Pi 5, laptops, cloud CPUs, GPUs, and the Qualcomm Hexagon NPU.

LFM3-1.7B-Base contains 24 layers and includes four main architectural changes:

1. Tri-hybrid layer mix: The model uses 13 gated short-convolution layers, 5 sliding-window attention (SWA) layers with a 256-token window, and 6 global-attention layers. The convolutions use a kernel size of 7, compared with 3 in LFM2.
2. Head-wise gated attention: A sigmoid gate scales the output of each attention head, allowing a head to switch off when it has nothing to contribute. This can help suppress activation outliers that interfere with quantization.
3. Partial RoPE: In global-attention layers, rotary positional embeddings rotate only the highest-frequency half of each head. The other half therefore matches content regardless of distance. SWA layers retain full RoPE.
4. Sandwich normalization: Each block normalizes both its input and output to keep activations well-scaled.

The architecture shows its benefits on long contexts. Per parameter, LFM3-1.7B matches LFM2-1.2B on short prompts across edge devices and is about 25% faster with 8K tokens in context.

LFM3-1.7B-Base was pretrained on 22T tokens using the same 128K vocabulary as LFM2.5-8B-A1B. Its final 3.5T tokens came from a mid-training stage using curated data with a larger proportion of agentic traces and tool calls. This stage also extended the context window to 256K tokens.

One backbone for six models

Every LFM3 model branches from LFM3-1.7B-Base. Instead of using one large general-purpose model for every workload, developers can select the model that matches their inputs, customize it, and run it on a local device.

The vision and audio models also use two in-house encoders:

• Vision encoder: This encoder combines convolution and attention and uses 3x less compute than the SigLIP2 encoder used in earlier LFM VL models. Its 164M parameters match the accuracy of Qwen3.5-2B's vision encoder while requiring 2.2x fewer FLOPs. Both measurements use the same LFM2.5-230M language backbone. For video, each pair of sampled frames is embedded into one temporal slice, reducing the temporal sequence length by half.
• Audio encoder and LiquidCodec: The audio encoder combines sliding-window attention with short convolutions and is fully causal. This allows the model to prefill while a person is speaking, with time to first token as low as 50 ms. LiquidCodec, the new audio tokenizer, converts generated tokens into speech at about 87x real time on a single CPU thread on Apple M4 Max and supports streaming decoding.

Using the LFM3 models

Each model runs in transformers. The following examples show basic usage.

LFM3-1.7B

LFM3-1.7B is intended for local agents, tool calling, structured outputs, and domain-specific fine-tuning. It is post-trained for agentic harnesses such as Hermes Agent, Pi, and OpenClaw.

The model outperforms LFM2.5-2.6B with 35% fewer parameters. Its largest gains are on agentic tasks. On WebArena-Verified, it scores 38.42 compared with 3.02 for LFM2.5-2.6B, and it also exceeds Qwen3.5-9B on long tasks in browsers and business applications.

Install the latest version of transformers:

pip install -U transformers

Load and run the model:

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="LiquidAI/LFM3-1.7B",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "What is C. elegans?"}]
output = generator(
    messages,
    do_sample=True,
    temperature=0.1,
    top_k=50,
    repetition_penalty=1.1,
    max_new_tokens=2048,
)
print(output[0]["generated_text"][-1]["content"])

LFM3-VL-2B

LFM3-VL-2B provides on-device visual understanding. It is the first LFM vision model to accept video input. The model reasons step by step before answering, detects objects, and reasons about their spatial arrangement, making it suitable as a foundation for robotics applications.

Compared with LFM2.5-VL-3B, LFM3-VL-2B has fewer parameters, 1.9B versus 3.1B, while increasing the average score across 62 vision benchmarks from 57.6 to 71.4. This is on par with Gemma 4 12B, which has more than six times as many parameters.

Install the required dependencies:

pip install "transformers>=5.21.0" torch torchvision pillow

The following example processes an image and generates a description:

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "LiquidAI/LFM3-VL-2B"

model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://placecats.com/300/200"},
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.1,
        top_k=50,
        repetition_penalty=1.1,
        max_new_tokens=16384,
    )

generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])

LFM3-Audio-2B

LFM3-Audio-2B supports voice chat, speech recognition, speech synthesis, and translation across 18 languages on laptops and edge devices. It accepts text or audio and can respond with text, speech, or both. Spoken requests can also be converted into tool calls.

Compared with LFM2.5-Audio-1.5B, it uses a new in-house audio encoder and speech codec and produces higher-quality speech, with a UTMOSv2 score of 3.83 compared with 3.22. It also adds audio understanding, general-purpose audio reasoning, and tool calling.

Install the dependencies:

pip install "transformers>=5.20.2" torch accelerate librosa soundfile

Load the model, processor, and codec:

import torch
import soundfile as sf

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "LiquidAI/LFM3-Audio-2B"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

model = AutoModelForMultimodalLM.from_pretrained(
    model_id, dtype=dtype, device_map=device, trust_remote_code=True
).eval()
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
codec = processor.audio_tokenizer.to(device=device).eval()

The following example accepts a spoken question and produces text and speech:

conversation = [
    {"role": "system", "content": "Respond with interleaved text and audio."},
    {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]},
    {"role": "assistant", "content": "<|mixed_start|>"},
]
inputs = processor.apply_chat_template(
    conversation,
    continue_final_message=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        audio_temperature=1.0,
        audio_top_k=128,
        audio_repetition_penalty=1.1,
        audio_repetition_window=4096,
        return_dict_in_generate=True,
    )

    text = processor.decode(
        outputs.sequences[0, inputs["input_ids"].shape[1]:],
        skip_special_tokens=True,
    )
    waveform = codec.decode(
        outputs.audio_codes[:1, :, :outputs.audio_lengths[0]]
    ).audio_values[0]
print(text)
sf.write("reply.wav", waveform.float().cpu().numpy(), 24_000)

LFM3-Omni-2B

LFM3-Omni-2B is an on-device model for applications that need to see, listen, and respond. It accepts text, images, video, and audio, and produces text or speech. It can answer spoken questions about images, process video and its soundtrack together, and convert visual or audio observations into tool calls.

The model is the first LFM omni model and retains the main capabilities of the standalone LFM3 models, including visual question answering, speech recognition, speech synthesis, and instruction following. Like LFM3-Audio-2B, it uses LiquidCodec for speech output. LiquidCodec converts generated tokens into speech faster than real time, allowing spoken responses to stream as they are generated.

On audio-visual video understanding, LFM3-Omni-2B outperforms models with up to 10.7B parameters, including Qwen2.5-Omni-7B, on Daily-Omni, WorldSense, and AV-SpeakerBench.

Install the required dependencies:

pip install "transformers>=5.20.2" torch torchvision torchcodec accelerate pillow librosa soundfile

Load the model and processor:

import torch
import soundfile as sf

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "LiquidAI/LFM3-Omni-2B"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

model = AutoModelForMultimodalLM.from_pretrained(
    model_id, dtype=dtype, device_map=device, trust_remote_code=True
).eval()
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
codec = processor.audio_tokenizer.to(device=device).eval()

The following example asks a spoken question about a photograph:

conversation = [
    {
        "role": "system",
        "content": "Answer questions about the provided image. Base every answer only on what can be seen. Respond with text.",
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "photo.jpg"},
            {"type": "audio", "audio": "question.wav"},
        ],
    },
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text")

print(processor.decode(
    outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]],
    skip_special_tokens=True,
))

d1-2B and d1-omni-2B

The d1 models are designed for fast, structured decisions rather than generated prose. They can process a situation such as a support ticket, invoice, application screen, photograph, or voice note, then answer several questions in a single forward pass. Each answer includes a calibrated probability.

In addition to yes/no questions, choices, and scores, the models can point to objects, read values from documents, estimate numbers with a range, and select every applicable option.

• d1-2B: Supports text, images, and video. It replaces d1-3B with one-third fewer parameters and scores higher on all four decision suites. On ImajevBench, it scores 75.5 compared with 64.0 for d1-3B.
• d1-omni-2B: Supports audio as an input. When questions are spoken rather than written, it retains nearly all of its accuracy.

Install the dependencies for d1-2B:

pip install "transformers>=5.19" torch torchvision pillow

The following example asks several questions about one support ticket in a single pass:

import torch
from transformers import AutoModel

device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
dtype = torch.float32 if device == "cpu" else torch.bfloat16
model = AutoModel.from_pretrained(
    "LiquidAI/d1-2B", trust_remote_code=True, dtype=dtype
).to(device)

questions = {
    "refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"},
    "team": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
            "billing": "Charges, refunds, invoices",
            "shipping": "Delivery and parcels",
            "technical": "App or site faults",
        },
    },
    "topics": {
        "type": "choice",
        "multiple": True,
        "instructions": "Which topics does the message raise?",
        "criteria": {
            "billing": "Payments and charges",
            "delivery": "Shipping and parcels",
            "login": "Signing in to the account",
        },
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgent is this ticket?",
        "criteria": ["low", "medium", "high"],
    },
}
out = model.system_one(
    "I was charged twice and my parcel never arrived. Please refund one charge.",
    questions,
)
a = out["answers"]
urgency = a["urgency"]["legend"][str(int(a["urgency"]["score"]))]
print(
    round(a["refund"]["noul"], 2),
    a["team"]["choice"],
    a["topics"]["choices"],
    urgency,
)
## 1.0 billing ['billing', 'delivery'] medium

Fine-tuning LFM3 for specialized tasks

LFM3 is intended to be specialized for particular datasets, terminology, and tools. Its relatively small size allows fine-tuning on a single GPU and deployment on devices, private infrastructure, or the cloud.

The available notebooks cover data preparation, training, and evaluation for text, vision, audio, and omni models:

Example: adapting LFM3 for robotics

LFM3-VL-2B can be adapted into three complementary robotics models for orchestration, navigation, and manipulation.

The orchestrator operates a robot arm through tool calls. It grounds objects by pointing, measures them with depth tools, invokes motion planners, and checks results in subsequent camera images.

The navigator follows route descriptions using recent RGB observations. It predicts spatial targets and a waypoint trajectory in five tokens.

The manipulation vision-language-action model maps images, language, and robot state to eight-step chunks of continuous actions in one forward pass. Trained across 15 embodiments, the VLA achieves 96.2% average success across the four LIBERO suites and 45.7% on RoboCasa-GR1 tabletop tasks. It generates action chunks in under 50 ms on a Jetson AGX Orin with TensorRT FP16.

These results show how LFM3-VL-2B can serve as a compact robotics foundation, covering high-level planning and low-level control with real-time action generation on embedded hardware.

LFM3 model availability

All LFM3 models are open-weight and available on Hugging Face. The collection includes LFM3-1.7B-Base for custom post-training, along with the text, vision, audio, omni, and decision models described above.

Fine-tuning notebooks cover the supported training methods and modalities, while the models can also be used with local agent harnesses such as OpenClaw, Hermes Agent, and Pi.