Phonon TTS: High-Quality Speech Generation on the Device

Phonon TTS: High-Quality Speech Generation on the Device

Phonon is a compact text-to-speech model designed to run locally on a device without a server or network connection. This article explains the model's design, available releases, early applications, leaderboard results, and inference options.

Why run speech generation locally?

Many voice AI systems operate in the cloud, sending each sentence to a remote server for processing. Phonon takes a different approach: the text remains on the user's device, and speech generation continues without a network connection.

The model weights are available under different licenses. Phonon 31m uses Apache 2.0, allowing free use in commercial products. The other models use CC BY-NC and are available for non-commercial projects. Commercial pricing for those models is based on the number of users rather than the amount of generated speech.

Running a high-quality voice model on a small device has traditionally required a compromise between model size and audio quality. Phonon was developed to address both constraints. The project was announced in April, followed by six months of development with partners and early users. That work led to smaller models, support for additional languages, and custom voices through voice cloning and voice design.

Phonon follows the Pocket TTS lineage and uses flow matching to generate audio. This differs from the techniques used by many other TTS systems and enabled the model to provide its quality at a small size. Training introduced numerical-stability challenges, requiring several techniques to produce consistent results across all five languages.

Released models and capabilities

Phonon is available in two sizes, approximately 65 MB and 75 MB. Across the releases, it supports English, French, German, Spanish, and Portuguese. The models include more than 100 built-in voices, and custom voices can be created in Gradium Studio.

The system is designed to handle inputs such as phone numbers, dates, and codes, which can be difficult for speech systems to pronounce correctly. It runs on phones, laptops, single-board computers, and in a browser tab.

Phonon 31m supports English and is released under Apache 2.0, including for commercial applications. Phonon 40m supports all five languages under a non-commercial license.

Demonstrations and early integrations

The launch included several demonstrations and early projects:

• An Apple Watch with Wi-Fi disabled provides step-by-step guidance for brewing coffee.
• An iPhone without a cellular signal counts exercise repetitions and provides encouragement.
• Autonomous's Lamp robot uses a sorting-hat character and speaks in character.
• An offline dinosaur runner in a browser becomes a radio show hosted by Rex.
• A science-fiction game generates speech locally on the player's machine.
• Proust is read aloud page by page.

The Hugging Face team added Transformers integration so that developers could use Phonon in Transformers from its initial release.

Other integrations include an ExecuTorch conversion that runs Phonon on a visitor's browser GPU, Micdrop support for Node.js applications, and Spaces such as smallest-s2s.

Phonon was downloaded more than 200,000 times during its first two weeks and continued to trend on Hugging Face during that period.

Results on the Hugging Face Open TTS Leaderboard

The Open TTS Leaderboard provides a way to compare open speech models and listen to their outputs. The Hugging Face Hub contains more than 9,000 TTS models. The leaderboard evaluates models across 23 languages using several measures:

• Intelligibility: Word error rate of the generated speech.
• Speed: Inverse real-time factor and time to first audio.
• Naturalness: UTMOS for English and KOMOS for Korean.
• Speaker similarity: An additional measure for voice-cloning evaluations.

The evaluations cover both fixed-voice generation and voice cloning. A Listen tab also allows users to compare outputs and vote for their preferred results.

Phonon ranks first for intelligibility in German and Spanish. It places in the top three for every language evaluated, alongside models that can be up to 100 times larger.

The model's strongest leaderboard result is speed. Phonon leads the CPU streaming benchmark for time to first audio. In the 95th-percentile results, Phonon 31m en reaches the first audio in 28 ms, while Phonon 40m en reaches it in 38 ms. Pocket TTS records 165 ms in the same comparison, where lower is better.

Inference frameworks

A dedicated inference framework called ptts was developed for Phonon. It supports more than 10 backends across different device types:

• CPU: Intel, Arm, and WebAssembly
• GPU: CUDA, Vulkan, Metal, and WebGPU
• NPU: Apple Core ML and Snapdragon®

Phonon is also available in Transformers from version 5.21 as a reference PyTorch implementation. The two approaches target different workloads. ptts runs a single stream on a device, while Transformers supports GPU throughput through batched generation and full-graph torch.compile, including speech dataset synthesis at scale. Its native PyTorch implementation can also be used to integrate or study the model.

Because Phonon belongs to the Pocket TTS lineage, Pocket TTS was first added to Transformers. Phonon is implemented as a modular model, with about 200 lines describing the parts that differ.

Audio latents and tokenization

Phonon generates audio latents autoregressively. Each generated latent is fed back as the next input, and the training loss is calculated over the latent sequence. The latents function as inputs, outputs, and training targets, much as text tokens do in a language model. The codec that maps audio to latents therefore serves a role similar to a tokenizer.

Transformers exposes this component as an audio tokenizer, stored by the processor alongside the text tokenizer:

from transformers import AutoProcessor, GradiumPhononTTSForConditionalGeneration

model_id = "gradium/phonon-31m-en"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = GradiumPhononTTSForConditionalGeneration.from_pretrained(model_id, device_map="auto")

print(f"{model.num_parameters():,}")  # 30,788,354
print(f"{processor.audio_tokenizer.num_parameters():,}")  # 10,305,313

A voice is represented as a tensor of speaker latents rather than as a recording. Phonon does not encode audio directly with its audio tokenizer, which is distributed as decoder-only. Voices are provided as precomputed files in gradium/phonon-voices:

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

path = hf_hub_download("gradium/phonon-voices", "voices/LMDIYE0Oa377Qh6Z.safetensors")
voice = load_file(path)["speaker_wavs"][0].T  # (num_frames, 512)

Preparing a prompt

The prompt layout contains voice positions, text, and an audio-opening token. This layout is included with the checkpoint as a chat template, as with other multimodal models in Transformers. The conversation uses an empty audio item as a placeholder for the voice:

conversation = [
    {"role": "user", "content": [{"type": "audio"}, {"type": "text", "text": "Hello from Phonon."}]}
]
inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    return_dict=True,
    add_generation_prompt=True,
    processor_kwargs={"audio_prompt_latents": [voice]},
).to(model.device)
## prompt: <|audio_prompt|> × num_frames, "Hello from Phonon.", <|audio_bos|>

The processor expands the voice placeholder to one position per frame, normalizes and tokenizes the text, and batches the inputs. The model then generates audio latents, which the processor decodes into audio:

outputs = model.generate(**inputs)  # audio_latents: (batch_size, num_frames, 32)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(audio, "phonon.wav")

Future directions

Phonon can use custom voices created in two ways. A voice can be cloned from a short recording with the speaker's consent, or designed by describing it in plain text in Gradium Studio. The resulting voice file can then be loaded into Phonon for fully local inference.

Custom Phonon models are also trained with design partners. These models can be adapted to tighter memory limits, additional languages such as Korean, Arabic, or Japanese, and product-specific vocabulary through fine-tuning.