Muse Spark Open: Meta’s Open-Weight Multimodal Mixture-of-Experts Model
Muse Spark Open: Meta’s Open Weight Multimodal Mixture of Experts Model Published October 15, 2026 Muse Spark Open is an open weight model from Meta that processes text, images,...
By Software Development Team
Muse Spark Open: Meta’s Open-Weight Multimodal Mixture-of-Experts Model
Published October 15, 2026
Muse Spark Open is an open-weight model from Meta that processes text, images, audio, and video in a single system. It uses a mixture-of-experts architecture with approximately 816 billion total parameters, about 42 billion active parameters per token, and an inference context window of up to one million tokens. The model generates text through a common interface for questions involving different input types.
Following Muse Glimmer, Muse Spark Open applies a mixture-of-experts design at a much larger scale. Meta trained it for agentic coding tasks, including reading codebases, writing and debugging code, calling tools, and working through long sequences of planning, action, observation, and self-correction. Because it accepts visual and audio inputs, the model can also inspect screenshots, listen to voice notes, and analyze screen recordings while working on code.
Capabilities and architecture
Meta describes Muse Spark Open as a coding-oriented model trained on developer workflows such as multi-file refactoring and extended debugging sessions. Its main specifications are:
| Feature | Muse Spark Open |
|---|---|
| Model size | Approximately 816B total parameters, approximately 42B active per token |
| Expert routing | 256 experts per MoE layer, with 8 selected per token |
| Inputs | Text, images, audio, and video |
| Output | Text |
| Context window | Up to 1M tokens |
| Checkpoint formats | BF16 and NVFP4 |
The checkpoint repositories are Muse-Spark-Open-816B-A42B and Muse-Spark-Open-816B-A42B-NVFP4.
Muse Spark Open is the full-size model in the Muse series. Muse Glimmer, released by Meta in August, is a smaller dense model intended for local execution. Both models use the same tokenizer and vocabulary. Muse Spark Open follows the Muse Glimmer chat template, with additions for image, audio, and video inputs. Its input embedding and output layers are separate matrices of approximately 1.66 billion parameters each.
Mixture of experts
The language model contains 62 layers with a hidden dimension of 8,192. Each layer has 128 query heads and 16 key-value heads using grouped-query attention. Every head has 64 dimensions. The attention block includes an output gate, output normalization, and query-key normalization, or QK-norm. Each block combines its input and output through a learned residual gate rather than a simple sum.
Muse Glimmer uses one dense feed-forward network per layer. Muse Spark Open replaces that network with a mixture-of-experts block containing 256 experts. The router selects 8 experts for each token. Each expert projects from 8,192 to 4,096 dimensions and back to 8,192, then applies its own output normalization. The experts contain approximately 97% of the model’s total parameters.
Only 8 experts run for each token, so computation uses approximately 42 billion parameters per token. However, all 256 experts must remain in memory. Deployment therefore requires memory for an 816-billion-parameter model while providing computation closer to a 42-billion-parameter model.
The BF16 checkpoint is approximately 1.5 TB in safetensors format. The NVFP4 checkpoint stores parameters at 4-bit precision with negligible accuracy loss and is designed for NVIDIA Blackwell GPUs. In Transformers, the BF16 checkpoint runs across four nodes with eight H200 GPUs each using tensor parallelism with TP=32. The NVFP4 checkpoint runs on a single B200 node. Deployment options include llama.cpp, vLLM, and SGLang.
Perception encoder
The perception encoder converts images and video frames into tokens for the language model. It has 1.8 billion parameters, 50 layers, a hidden dimension of 1,536, and 16 attention heads. Images are divided into 14×14 patches, producing features with 4,096 dimensions.
Position information comes from learned absolute embeddings and two-dimensional RoPE. Each feed-forward network uses LayerNorm and GELU. Every fourth layer and the final layer use full attention, while the remaining layers use sparse attention.
Muse Glimmer 30B uses an encoder with the same architecture. The design follows Meta’s Perception Encoder work. Video input uses the same encoder as image input.
Audio encoder
Muse Spark Open processes audio through an audio encoder. An audio tokenizer converts the waveform into tokens, and the chat template marks the audio position with <|audio|> tokens. The encoder emits one audio token every 80 milliseconds, or 12.5 tokens per second. A 10-second recording therefore produces 125 audio tokens.
The encoder is a convolutional pyramid. Each stage contains a patch embedder with a different patch size and two convolutional blocks using one-dimensional kernels of size 7 with stride 1. The stages use patch sizes of 16, 6, 5, and 4. The final temporal features pass through an RMSNorm, linear projection, and another RMSNorm.
Long context
Muse Spark Open supports a one-million-token context during inference. Its configuration sets max_position_embeddings to 1,048,576. Image, video, and audio tokens share this context window with text.
Grouped-query attention reduces the size of the key-value cache. Each token adds:
2 × 16 heads × 64 dimensions × 62 layers = 126,976 values
In BF16, this is approximately 250 KB per token, or about 266 GB for a full one-million-token context.
Inference support
Transformers
The Transformers implementation provides MuseSparkForConditionalGeneration for text generation from multimodal inputs and MuseSparkProcessor for preparing conversations. The processor combines the tokenizer, image and video processors, audio feature extractor, and chat template.
A DistributedConfig controls parallelism. tp_size specifies the number of devices across which dense layers are sharded, while ep_size specifies the number of devices across which experts are distributed.
import os
from transformers import AutoModelForMultimodalLM, AutoTokenizer, DistributedConfig
distributed_config = DistributedConfig(
tp_size=int(os.environ["WORLD_SIZE"]),
ep_size=int(os.environ["WORLD_SIZE"]),
)
tokenizer = AutoTokenizer.from_pretrained(
"meta-models/Muse-Spark-Open-816B-A42B"
)
model = AutoModelForMultimodalLM.from_pretrained(
"meta-models/Muse-Spark-Open-816B-A42B",
distributed_config=distributed_config,
)
messages = [
{"role": "user", "content": "Explain in one sentence why the sky is usually blue."},
]
inputs = tokenizer.apply_chat_template(
messages,
reasoning_effort="medium",
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device).to(model.dtype)
outputs = model.generate(**inputs, max_new_tokens=2000)
Text-only inference
For text-only inference, pass messages to processor.apply_chat_template with add_generation_prompt=True, tokenize=True, return_dict=True, and return_tensors="pt". The resulting inputs can be passed to model.generate.
The processor supports symbolic reasoning levels of low, medium, high, and xhigh, corresponding to the integer values 16, 64, 256, and 512 accepted by the model. Reasoning cannot be disabled, so low is the appropriate setting when an easier task requires fewer generated tokens. A numeric level can also be specified in the system prompt, for example, Reasoning strength: 1.
messages = [
{"role": "system", "content": "You should only answer with a number."},
{"role": "user", "content": "What is 17 * 23?"},
]
inputs = processor.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
reasoning_effort="high",
).to(model.device).to(model.dtype)
output = model.generate(**inputs, max_new_tokens=2000)
generated_tokens = output[0][inputs["input_ids"].shape[1]:]
print(processor.decode(generated_tokens, skip_special_tokens=False))
Images and text
The processor expands each image marker into the required number of patch tokens and surrounds the expansion with image boundary tokens. This preserves alignment between text positions and visual features. Multiple images can be included in one prompt.
image_url = (
"https://huggingface.co/datasets/merve/vl-test-suite/"
"resolve/main/pills.jpg"
)
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image_url},
{
"type": "text",
"text": "Do any of the components in this supplement interact?",
},
],
},
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
reasoning_effort="medium",
return_dict=True,
return_tensors="pt",
).to(model.device).to(model.dtype)
input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs, max_new_tokens=2000)
response = processor.decode(
outputs[0][input_len:],
skip_special_tokens=False,
)
processor.parse_response(response)
Video inference
Video prompts include frame timing alongside visual content. The processor can handle videos that contain audio, an improvement over Muse Glimmer made possible by the audio encoder. It downsamples video frames and audio so that they remain aligned.
By default, videos are sampled at 2 FPS with a maximum of 96 frames. The frame rate can be changed through the video processor:
processor.video_processor.fps = 4.0
Higher frame rates preserve more events but require additional memory and increase prefill latency. Frames are interleaved with text containing timestamps such as Time: 3.5s. To include a video’s audio, pass load_audio_from_video=True to apply_chat_template. Using return_metadata=True preserves video metadata so the processor can assign the correct frame rate.
import torch
messages = [
{
"role": "user",
"content": [
{
"type": "text",
"text": (
"Describe what happens in this video. "
"Use timestamps and include relevant speech or sounds."
),
},
{
"type": "video",
"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/ramen.mp4",
},
],
}
]
inputs = processor.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
load_audio_from_video=True,
return_metadata=True,
).to(model.device).to(model.dtype)
with torch.inference_mode():
output_ids = model.generate(
**inputs,
max_new_tokens=256,
do_sample=False,
)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
response = processor.batch_decode(
generated_ids,
skip_special_tokens=True,
)[0]
print(response)
Audio inference
The audio tower expects a mono waveform sampled at 24 kHz. The processor pads audio to a multiple of 1,920 samples, which corresponds to one audio embedding every 80 milliseconds. Audio at another sample rate must be resampled first. Transformers’ load_audio utility can perform this using torchcodec, librosa, or torchaudio backends.
When audio is supplied through apply_chat_template, downloading, decoding, resampling, padding, mask creation, and placeholder expansion are handled by the processor.
import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM
model_id = "meta-models/Muse-Spark-Open-816B-A42B"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id).eval()
messages = [
{
"role": "user",
"content": [
{
"type": "audio",
"url": "https://huggingface.co/buckets/huggingface/audio-samples/resolve/bbaudio_question_39.mp3",
},
{"type": "text", "text": "Answer the question."},
],
}
]
inputs = processor.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
load_audio_backend="auto",
).to(model.device).to(model.dtype)
with torch.inference_mode():
output_ids = model.generate(
**inputs,
max_new_tokens=128,
do_sample=False,
)
prompt_length = inputs["input_ids"].shape[1]
generated_text = processor.batch_decode(
output_ids[:, prompt_length:],
skip_special_tokens=True,
)
print(generated_text[0])
The processor inserts the corresponding number of audio tokens between audio boundary markers. It also returns a mask that identifies real samples so padded audio embeddings can be excluded.
Tool calling
Muse Spark Open supports text-only and multimodal tool calling. Tools are passed to apply_chat_template along with the conversation.
from transformers import AutoProcessor, AutoModelForMultimodalLM
MODEL_ID = "meta-models/Muse-Spark-Open-816B-A42B"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
MODEL_ID,
dtype="auto",
device_map="auto",
)
tools = [
{
"type": "function",
"function": {
"name": "weather.get",
"description": "Get the current weather for a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}
]
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png",
},
{
"type": "text",
"text": "I'm going to the city in this picture. What clothes should I wear?",
},
],
},
]
inputs = processor.apply_chat_template(
messages,
tools=tools,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
reasoning_strength="high",
).to(model.device).to(model.dtype)
outputs = model.generate(**inputs)
response = processor.parse_response(
outputs,
prefix=inputs["input_ids"][0],
tools=tools,
)
print(response)
Muse Spark Open uses a channel-based chat template called Harmony, first introduced by gpt-oss. The assistant channel is left open so the model can select a recipient with to=. The main recipients are:
self, for internal reasoningcommentary, for comments before or between tool callsuser, for the final answer
A tool call may be represented as follows:
<|start|>assistant to=weather.get<|message|>
<atem:function_calls>
<atem:invoke name="weather.get">
<atem:parameter name="city">San Francisco</atem:parameter>
</atem:invoke>
</atem:function_calls>
<|eot|>
Llama.cpp
Muse Spark Open is integrated into llama.cpp and is available in its prebuilt binaries. Installation can be performed with:
curl -LsSf https://llama.app/install.sh | sh
A server can then be started with a quantization suffix:
llama serve -hf ggml-org/Muse-Spark-Open-816B-A42B-GGUF:Q2_K
The built-in WebUI is available at localhost:9931 after startup. It supports image and audio inputs and provides a reasoning-level control.
Tests on a 512 GB Mac M3 Ultra measured 18 tokens per second for an imatrix-calibrated 2-bit quantization. The model used 304 GB, leaving space for the KV cache.
The server exposes an OpenAI-compatible API:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a limerick about python exceptions"}
]
}'
llama.cpp provides several quantized versions. Q2_K, Q4_K_S, and Q8 were tested on Apple Silicon. A quantization can be selected by adding its identifier as a suffix, such as Q2_K.
SGLang
The SGLang integration supports text, image, audio, and video inputs. The NVFP4 reference configuration runs across four GB300 GPUs.
export MODEL_ID=meta-models/Muse-Spark-Open-816B-A42B-NVFP4
export CKPT="$(hf download "$MODEL_ID")"
SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=1 \
uv run sglang_meta serve \
--model-path "$CKPT" \
--served-model-name "$MODEL_ID" \
--quantization modelopt_fp4 \
--moe-runner-backend flashinfer_trtllm \
--tp-size 4 \
--ep-size 4 \
--chat-template "$CKPT/muse_spark_1_2_oss.jinja" \
--port 30000
The checkpoint includes the chat template, which is passed explicitly in the command. The launcher configures reasoning and tool-call parsers. Requests can be sent through the OpenAI-compatible API:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
"messages": [{
"role": "user",
"content": "Explain how a mixture-of-experts model works."
}],
"max_tokens": 2048
}'
The response places the reasoning trace in reasoning_content and the final answer in content.
vLLM
The vLLM reference integration supports text and image inputs with the NVFP4 checkpoint on Blackwell-series GPUs.
export MODEL_ID=meta-models/Muse-Spark-Open-816B-A42B-NVFP4
export CKPT="$(hf download "$MODEL_ID")"
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_PLUGINS=musespark1x_models,ptq_nvfp4 \
vllm serve "$CKPT" \
--served-model-name "$MODEL_ID" \
--tokenizer-mode musespark1x \
--config-format auto \
--load-format internal_safetensor \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--max-model-len 131072 \
--gpu-memory-utilization 0.92 \
--reasoning-parser musespark1x \
--tool-call-parser musespark1x \
--enable-auto-tool-choice \
--port 8000
This configuration sets a 128K-token serving context. The plugin supplies the model implementation, checkpoint loader, and reasoning and tool-call parsers.
MLX
Muse Spark Open is available in oMLX and mlx-lm. oQ quantizations optimized for oMLX are available in 3-bit and 4-bit formats. Testing of the oQ3e quantization on a 512 GB M3 Ultra used approximately 344 GB, with measured performance of about 225 tokens per second during prefill and 21 tokens per second during generation using a 1,024-token prompt and 128 generated tokens.
The oMLX chat interface supports images and adjustable reasoning effort from low to xhigh. Reasoning appears in a collapsible block separate from the final answer. Reasoning cannot be disabled, and the default Auto setting uses xhigh. The low setting reduces token usage for easier tasks.
The model is also available in a pull request for the official mlx-lm repository:
mlx_lm.generate --model mlx-community/Muse-Spark-Open-816B-A42B-oQ3e \
--prompt "Explain convolutional networks like I am 5" \
--max-tokens 5000
Even the 3-bit quantization requires more than 256 GB of RAM. Distributed inference across multiple computers is also supported.
Inference Endpoints
Muse Spark Open can be deployed on Inference Endpoints with a pre-validated SGLang recipe for the NVFP4 variant. The configuration runs on either eight NVIDIA RTX Pro 6000 98 GB GPUs with a 131,072-token context, or eight NVIDIA H200 141 GB GPUs for a 1,048,576-token context.
Deployment can be started with the Hugging Face CLI:
hf endpoints catalog deploy \
--repo meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
--name muse-spark
The endpoint exposes OpenAI-compatible Chat Completions and Responses APIs. A request can include multimodal content, such as an image and text, through /v1/chat/completions.
Fine-tuning with TRL
The full 816-billion-parameter checkpoint can be fine-tuned with the TRL SFTTrainer. One demonstrated recipe trained LoRA adapters on the attention layers while keeping the experts frozen. Training used eight nodes with eight H100 GPUs each.
The model was split across 64 GPU devices using tensor, data, and expert parallelism through DistributedConfig:
from transformers.distributed import DistributedConfig
distributed_config = DistributedConfig(fsdp_size=64, ep_size=8)
Here, fsdp_size=64 shards the model across all 64 GPUs with FSDP2, while ep_size=8 divides the 256 experts in each layer into eight groups. Each token is routed to the GPU containing its selected expert.
import torch
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
train_dataset = load_dataset(
"allenai/tulu-3-sft-mixture",
split="train[:500000]",
)
training_args = SFTConfig(
output_dir="muse-spark-lora-sft",
model_init_kwargs={
"dtype": torch.bfloat16,
"distributed_config": distributed_config,
},
per_device_train_batch_size=1,
max_steps=500,
max_length=2048,
packing=True,
save_only_model=True,
logging_steps=1,
report_to="trackio",
)
trainer = SFTTrainer(
model="meta-models/Muse-Spark-Open-816B-A42B",
args=training_args,
train_dataset=train_dataset,
peft_config=LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
),
)
trainer.train()
packing=True joins examples into 2,048-token sequences without padding. A block-diagonal mask keeps the examples separate. save_only_model=True saves the adapter without optimizer state because the trainer cannot resume a model loaded in sharded form. LoraConfig targets the four attention projections, while the 256 experts in each layer remain frozen.
Training was launched across the eight nodes with:
torchrun --nnodes 8 --nproc_per_node 8 \
--rdzv_backend c10d \
--rdzv_endpoint "$MASTER_IP:29700" \
sft_muse_spark.py
The 500-step run took approximately two hours on 64 H100 GPUs and processed about 8,000 tokens per second. Mean token accuracy increased from 72% to 84%, while loss declined from 1.26 to 0.57.
Demonstrations
Muse Spark Open supports coding and agentic workflows that combine multimodal inputs with tool use.
3D asset generation
A 2-bit quantized Muse Spark Open model served with llama.cpp was combined with the Pi coding agent to generate and iteratively improve Three.js assets. On a Mac Ultra, the model produced approximately 18 tokens per second.
The workflow begins with a description of the desired asset. Pi asks the model to generate Three.js code, renders the result in a browser, and captures screenshots from multiple angles. Those screenshots are sent back to the same model for visual feedback. Pi then edits the code and repeats the process.
Browser interaction with OpenEnv and Hugging Face Sandboxes
A browser-use demonstration combines Muse Spark Open with OpenEnv and Hugging Face Sandboxes. The system contains three components:
- Muse Spark Open served through Inference Endpoints
- OpenEnv, which wraps browser-driver actions such as clicking, typing, and scrolling through an environment interface
- A Chromium browser
The remote sandbox contains the environment and browser. The model receives the previous actions, system prompt, URL, and browser screenshot. It does not receive the DOM or HTML source, making the workflow a multimodal browser interaction.
Voice-controlled browser interaction
Another demonstration combines the model’s audio, image, and agentic capabilities in a browser agent. The user provides a general description of a product through voice input. Muse Spark Open reasons about the requested product, searches Amazon through an integrated browser, and selects an option according to the provided criteria.
Conclusion
Muse Spark Open is the full-size open-weight model in Meta’s Muse series. It combines text, image, audio, and video input with a mixture-of-experts architecture containing approximately 816 billion parameters, 42 billion active parameters per token, and a context window of up to one million tokens. The model can be run through Transformers, llama.cpp, SGLang, vLLM, MLX, and Inference Endpoints, and can be fine-tuned with LoRA adapters through TRL.