AnySpeechRecognizer and AnyDecisionModel Bring Open Speech and Decision Models to Apple Platforms

AnySpeechRecognizer and AnyDecisionModel Bring Open Speech and Decision Models to Apple Platforms

Published October 29, 2026

Language models can write, summarize, translate, and call tools, but they are only one category of machine-learning system. A customer-support application, for example, may first need speech recognition and then a routing decision. Dedicated models can handle those tasks with lower latency and cost than a general-purpose language model.

Following the AnyLanguageModel 1.0 release, two Swift packages extend the same model-selection approach to speech and decision-making:

• AnySpeechRecognizer provides an interface based on Apple's speech-to-text API while allowing applications to use different recognition engines, including NVIDIA's Parakeet and Alibaba's Qwen3-ASR locally through Core ML. It also adds speaker diarization.
• AnyDecisionModel answers yes-or-no, multiple-choice, and rating-scale questions with typed results and probabilities. It derives answers from model probabilities rather than generated text and supports local models through MLX, Core AI, and llama.cpp, as well as Jev and Ollama decision models.

Both packages can download open models from the Hugging Face Hub and run them on device.

AnySpeechRecognizer

Apple platforms have included speech recognition APIs for years. macOS 26 and iOS 26 introduced SpeechAnalyzer, a newer speech-to-text API. Applications add modules such as SpeechTranscriber and SpeechDetector, provide a file or stream of audio buffers, and consume results through asynchronous sequences. AssetInventory handles model installation, allowing applications to control when downloads occur.

AnySpeechRecognizer preserves that general API while allowing the recognition engine to be selected:

- import Speech
+ import AnySpeechRecognizer

The package mirrors SpeechAnalyzer, SpeechTranscriber, DictationTranscriber, SpeechDetector, and AssetInventory. Apple's Speech framework remains the default engine. To use Parakeet, an application changes the transcriber configuration:

let transcriber = SpeechTranscriber(
    model: ParakeetSpeechModel(variant: .v3),
    transcriptionOptions: [],
    reportingOptions: [.volatileResults],
    attributeOptions: [.audioTimeRange]
)

Asset installation, analyzer execution, and result handling remain the same. Additional providers can be added by implementing SpeechRecognitionModel for configuration and SpeechRecognitionExecutor for a module's session.

Supported speech models

Parakeet v3 is NVIDIA's Parakeet TDT 0.6B v3. It recognizes 25 European languages and identifies the language automatically.

Parakeet Ultra is Moondream's post-trained version of v3. It produced more accurate results in most reported measurements.

Parakeet Unified is NVIDIA's English-only Parakeet.

All three Parakeet models use the package's Core ML runtime, which supports file and live input, volatile results, word timing, and word confidence.

Qwen3-ASR is Alibaba's speech model, available in 0.6B and 1.7B sizes. It recognizes 30 languages and 22 Chinese dialects, including Mandarin, Cantonese, Japanese, and Korean. The language can be detected automatically or specified explicitly:

let transcriber = SpeechTranscriber(
    model: QwenSpeechModel(
        variant: .large,
        language: Locale.Language(identifier: "yue")
    ),
    transcriptionOptions: [],
    reportingOptions: [],
    attributeOptions: []
)

Qwen3-ASR produces one final result for each segment of up to 40 seconds. It does not provide volatile results, word timing, or confidence values. The model runs on the GPU and is slower than Parakeet, making it more suitable for file transcription than live input.

Core ML models on the Hub

The models were converted to Core ML and published on the Hugging Face Hub:

Each repository records its source checkpoint and a checksum for every file. The package pins models to fixed revisions. Parakeet repositories also include an 8-second encoder for live input. It reduces live analysis time by about one-third, with a somewhat higher word error rate.

Most conversion workflows follow FluidInference's mobius scripts, and each conversion is checked against the original model.

Models download only when an application requests them. Each Parakeet model requires about 600 MB, the live encoder requires about 590 MB, Qwen3-ASR requires 0.9 to 2.4 GB, and Sortformer requires about 250 MB. Files use the standard Hugging Face cache, allowing a Mac application outside the sandbox to share them with Python and other Hugging Face tools.

Speaker diarization

Speaker diarization identifies who said each part of a conversation. Apple's Speech framework does not provide a diarization module, so AnySpeechRecognizer adds SpeakerDiarizer. It runs NVIDIA's Streaming Sortformer on Core ML alongside an Apple or Parakeet transcriber:

let diarizer = SpeakerDiarizer(model: SortformerDiarizationModel())

if let request = try await AssetInventory.assetInstallationRequest(
    supporting: [transcriber, diarizer]
) {
    try await request.downloadAndInstall()
}

let analyzer = SpeechAnalyzer(modules: [transcriber, diarizer])

The diarizer tracks up to four speakers, including overlapping speech. SpeakerTimeline collects the results and assigns each timed word to a speaker, making it possible to produce a transcript organized by speaker turns.

Sortformer v2 is the default. Version v2.1, trained with additional meeting data, can also be selected and produced more accurate meeting results in the reported measurements.

Performance measurements

The package was measured with its hear command-line tool on an M4 Max Mac Studio running macOS 27.0.1. The following word error rates were recorded on LibriSpeech files lasting 3 to 8 minutes:

Parakeet transcribes an hour of audio in about 20 to 40 seconds from a file, or about 1.5 to 2.5 minutes during live processing. After Core ML compiles the model, memory use is about 100 MB. Qwen3-ASR is considerably slower and requires a few gigabytes of memory.

Meetings remain difficult for all tested systems. On AMI recordings, where speakers often talk over one another, every tested engine had a word error rate above 23%.

The measurements came from one machine and a limited set of datasets. The hear evaluate command can be used with other audio, and the repository's natural-speech-eval.py tool reproduces the reported measurements. Testing so far covered macOS 27, not iPhone hardware.

AnyDecisionModel

AnyDecisionModel does not mirror an Apple framework. It provides a Swift interface for asking structured questions of a model and receiving typed answers with probabilities.

Applications commonly ask language models questions such as whether a message is urgent, which team should handle a ticket, or how frustrated a customer sounds. A typical implementation prompts a model and parses generated text. That produces an answer but does not expose the probabilities assigned to the alternatives.

AnyDecisionModel represents these operations directly in Swift:

import AnyDecisionModel

enum Department: String, Choosable {
    case billing, technical, sales
}

let ticket = "I was charged twice for my order, and now the app logs me out when I try to ask for a refund."

let model = MLXDecisionModel(
    modelID: "mlx-community/Qwen3-4B-Instruct-2507-4bit"
)
let session = DecisionSession(model: model, state: .text(ticket))

let urgent = try await session.probability(of: "Does this convey urgency?")
let team = try await session.choice("Which team?", from: Department.self)
let anger = try await session.score(
    "How frustrated?",
    levels: ["Calm", "Frustrated", "Very angry"]
)

urgent is a Double. team is a Choice<Department> containing the selected case, a probability for each option, and a confidence value. anger is a Score containing the expected level, a probability for each level, and a confidence value.

Because Department conforms to Choosable, team.value is a Department that can be used in an exhaustive switch statement.

The model-and-session pattern resembles Foundation Models and AnyLanguageModel. A DecisionSession contains one state, represented as text or JSON, and every question refers to that state. Questions are independent, so one answer does not become input to a later question. Multiple questions can be submitted together to decide(_:) as a batch.

For each question, the MLX backend creates a prompt with a one-token label for every permitted answer, then reads the probability of each label in a single forward pass.

Emoji Magnet

Emoji Magnet is a macOS application with a text field above a collection of 343 emoji. When a user enters a phrase such as "things you wear," matching emoji rise from the collection. Refining the phrase to "things you wear in winter" causes emoji that no longer match to fall back.

The application has no built-in keyword rules. When typing pauses, the model asks one yes-or-no question for each emoji, such as:

Statement: a heavy winter coat fits the description "things you wear". Is this statement true?

The session state contains the phrase. The demonstration uses the hosted Jev backend, which answers all 343 questions in one request and settles the emoji in about half a second for each phrase.

Without a TYPESAFE_AI_API_KEY, the application runs Qwen3 4B locally with the MLX backend. Prefix caching evaluates the state once, after which the backend answers 16 questions per forward pass. On an M4 Max, a group of 16 takes about 0.45 seconds after the model loads. Processing all 343 emoji takes about 10 seconds, with each group's results displayed as they arrive.

Decision heads and Clef

Reading next-token probabilities works with general-purpose language models, although those models were not specifically trained to make decisions. Cloudflare's Clef models use a trained decision head that accepts a state and a schema of typed questions. It returns probabilities for all permitted options across the questions in one forward pass. Clef-Flash is the 9B variant.

AnyDecisionModel can run Clef-Flash on a Mac with MLX:

let model = MLXHeadDecisionModel(
    modelID: "mlx-community/clef-flash-4bit",
    head: .jointSchema
)

The same sessions and questions are used, but MLXHeadDecisionModel reads decisions from Clef's trained decision head instead of next-token probabilities. The 4-bit model is a 6.2 GB download, and its model card recommends a Mac with at least 16 GB of memory. Clef supports images and video, but this backend accepts only text and JSON states.

Probability calibration and diagnostics

Raw language-model probabilities are often overconfident. Local backends accept a Calibration that applies a temperature to every answer and a bias to yes-or-no answers. Calibration should be fitted with one dataset and evaluated with a separate dataset.

Language models can also favor an option because of its position in a list. On the MLX, Core AI, and llama.cpp next-token backends, rotationDebiasing asks a choice question once for each rotation of the options and averages the results. This requires one prompt evaluation per rotation.

Each local choice and score includes a confidence value based on probability entropy. It measures how concentrated the probabilities are, not whether the answer is correct. Error rates on the target data should therefore be checked before selecting a threshold. Next-token backends also report how much probability the model assigned to the permitted answers. A low value indicates that most probability was assigned elsewhere.

Backends

AnyDecisionModel provides six backends:

• MLX runs a small language model on Apple silicon through mlx-swift-lm and downloads it from the Hub.
• MLX decision heads run models with trained decision heads, such as Clef.
• Core AI runs the same type of model as MLX through a Core AI model on macOS 27 and iOS 27.
• llama.cpp reads decisions from a local GGUF model through llama.swift.
• Jev calls TypeSafe's Jev through the System One API, or another compatible server such as Kev.
• Ollama calls a System One decision model on a local Ollama server or in Ollama's cloud. It requires Ollama 0.35.0 or later, or 0.40.5 or later for cloud models.

The next-token backends label answers as "yes" and "no," A through Z, or 0 through 9. Consequently, a choice can contain at most 26 options and a score can contain at most 10 levels. Decision heads do not have this limitation.

Like AnyLanguageModel, the package uses Swift package traits. Without traits, it has no third-party dependencies and provides the core types plus the Jev and Ollama backends, which also build on Linux. The MLX, CoreAI, or Llama trait enables the corresponding local backend.

Requirements and installation

Both packages require Swift 6.2 or later and macOS 26 or iOS 26 or later. The Parakeet and Sortformer runtimes, along with the MLX backend, require Apple silicon.

AnyDecisionModel's MLX build tasks require Xcode's Swift 6.4 or later and the Metal toolchain. Core AI requires Xcode 27 and an OS 27 runtime.

Packages can be added to Package.swift as follows:

dependencies: [
    .package(
        url: "https://github.com/mattt/AnySpeechRecognizer",
        from: "0.1.0"
    ),
    .package(
        url: "https://github.com/mattt/AnyDecisionModel",
        from: "0.6.0",
        traits: ["MLX"]
    ),
]

For now, AnyDecisionModel's CoreAI trait cannot be combined with AnySpeechRecognizer. coreai-kit 0.7.3 requires swift-huggingface below 1.0, while AnySpeechRecognizer requires the 1.x release.

AnySpeechRecognizer also includes the hear command-line tool for transcribing local files, labeling speakers, and evaluating results. Every hear command requires --backend apple, --backend parakeet, or --backend qwen. Before analysis, run hear prepare with the same backend and model options to install the required assets.

Links

• AnySpeechRecognizer on GitHub
• AnyDecisionModel on GitHub
• AnyDecisionModel-Examples on GitHub
• AnyLanguageModel on GitHub
• swift-huggingface on GitHub