Intelligence at Pocket Scale: Benchmarking Small Models on Mobile Phones

Intelligence at Pocket Scale: Benchmarking Small Models on Mobile Phones

A new benchmarking program evaluates both the capabilities and on-device performance of models designed to run on mobile phones. Intelligence is assessed through five evaluations covering instruction following, function and tool calling, knowledge, and reasoning. A separate inference benchmark, operated in partnership with Liquid AI, measures speed, latency, and memory use on physical phones in a standardized environment.

Both benchmark types use the same model builds, quantized to 4-bit or smaller. This reflects the precision at which small models are realistically deployed on mobile devices. A model qualifies for testing when it fits within 8 GB of memory after quantization, including the KV cache required for an 8K context.

Why Small Models Are Difficult to Benchmark

Models with fewer than a few billion parameters can follow instructions, call tools, and answer questions on mobile phones. However, published benchmark results often do not identify the exact quantized artifact, device, or runtime configuration that produced them. Several factors complicate comparisons in this area.

Frontier Evaluations Often Stop Discriminating

Many widely reported benchmarks were designed to distinguish frontier models. Small models frequently cluster near the bottom of these tests, providing little information about differences within the small-model category.

Evaluations of extended agentic work, including GDPval-AA v2 and Terminal-Bench v2.1, require long, multi-step tasks that small models rarely complete. As a result, they can produce near-zero scores for many models. Useful small-model benchmarks need evaluations that generate meaningful differences among models in this size range.

The Inference Stack Is Still Developing

Mobile runtimes are less mature than their datacenter counterparts, and configuration differences can affect results. Examples include chat templates that fail to render tool calls, parsers that discard parallel tool calls, and server flags that silently reduce the available context window.

Devices Change the Results

On-device performance depends on thermal conditions, power modes, cooling, and the way an operating system accounts for memory. Results are therefore meaningful only when reported with the exact model, quantization, runtime, configuration flags, and physical temperature controls used during testing.

A Model Name Can Represent Multiple Builds

The same model weights may be distributed in several quantization formats. Memory requirements vary according to quantization, architecture, context length, and KV-cache size. Quantized builds may be released by model developers or produced by third parties, with different levels of documentation and transparency.

What the Benchmark Includes

The program combines intelligence benchmarking administered by Artificial Analysis with inference benchmarking developed and operated by Liquid AI. Intelligence and inference are measured using the same 4-bit-or-smaller build that a device would load. The build is served with llama.cpp so the two measurements use closely aligned conditions.

Mobile Device Benchmark Set

The Mobile Device Benchmark Set contains five evaluations selected to represent capabilities commonly required from phone-scale models:

• BFCL: Tool calling across a 640-task subset covering tool selection, multi-turn tool use, and declining when no tool is appropriate.
• IFBench: Instruction following with precise, verifiable output constraints.
• AA-Omniscience: Knowledge accuracy and resistance to hallucination, divided equally between accuracy and non-hallucination.
• GPQA Diamond: Graduate-level scientific reasoning in a multiple-choice format.
• MATH-500: Competition mathematics evaluated through symbolic answer checking.

The overall score is a simple average of the five evaluations. The benchmark methodology documents the evaluation process and customizations to the underlying tests.

Mobile Phone Inference

Liquid AI developed Pipette, a mobile-device inference benchmarking product, and released it as open source on GitHub. The methodology uses a standardized testing harness and a climate-controlled physical testing facility to support repeatable measurements. The inference tests measure total generation time, output speed, and peak memory use.

Launch Results on the iPhone 17 Pro

At launch, Nanbeige4.2-3B and LFM2.5-2.6B shared the highest average score under a 16K context limit, with scores of 63. Ornith-1.0-9B scored 62, while Qwen3.5 9B (Reasoning) scored 61.

LFM2.5-2.6B reached its score with lower resource use. On an iPhone 17 Pro, it processed a standard 1,024-token prompt in 8.0 seconds and used 2.3 GB of memory. Nanbeige4.2-3B required 21.4 seconds and 4.0 GB. The two 9B models required more than 25 seconds and 6.9 GB. The benchmark set included 42 quantized builds, 34 of which ran successfully on the iPhone 17 Pro.

Speed and Intelligence

Six models were unbeaten on both intelligence and generation time on the iPhone 17 Pro:

• LFM2.5-230M: 27 points in 0.9 seconds
• MiniCPM5-1B: 45 points in 2.9 seconds
• LFM2.5-8B-A1B: 58 points in 5.7 seconds
• Ling 3.0 Tiny: 59 points in 5.7 seconds
• LFM2.5-2.6B: 63 points in 8.0 seconds
• Nanbeige4.2-3B: 63 points in 21.4 seconds

LFM2.5-8B-A1B and Ling 3.0 Tiny are mixture-of-experts models that activate approximately one billion parameters per token. This allows them to respond in under six seconds despite having 8B-class weights.

The Effect of the Context Limit

The 16K context limit affects the leaderboard. Qwen3.5 9B is highly capable but produces substantially more output during the evaluations than is practical for a mobile phone. It generated 74.5 million tokens during one pass of the benchmark set, compared with 5.2 million for Gemma 4 E4B (Reasoning), whose overall score was within two points.

Qwen3.5 9B (Reasoning) reached the 16K context limit in 29% of its generations, reducing its scores. With the limit increased to 64K, Qwen3.5 9B (Reasoning) took first place with a score of [XX], ahead of the 16K leaders Nanbeige4.2-3B and LFM2.5-2.6B. [XX] followed with [XX], and [XX] with [XX].

A 64K context window does not fit within the memory available on a mobile phone. Qwen3.5 9B (Reasoning) produced 12 output tokens per second on the iPhone 17 Pro. At that speed, filling even a 16K window would require more than twenty minutes.

A One-Minute Answer Budget

Limiting each answer to the number of tokens a model can generate in 60 seconds on the iPhone 17 Pro changes the ranking. This calculation uses output speed multiplied by 60 and ignores prompt-processing time.

Under that limit, LFM2.5-8B-A1B ranked first with 47 points, followed by LFM2-2.6B-Exp with 45 and Gemma 4 E4B (Non-reasoning) with 44. LFM2.5-2.6B fell to 38, while Nanbeige4.2-3B fell to 18. At 14 output tokens per second, Nanbeige4.2-3B can produce approximately 850 tokens per answer, leaving much of its reasoning unfinished. Qwen3.5 9B (Reasoning) scored 14.

The one-minute limit is not the primary filter for the benchmark, but it represents a different operating constraint for mobile applications, where extended waiting times may not be practical.

Different Capability Profiles

The leading models show different strengths across the individual evaluations:

• Nanbeige4.2-3B is the most balanced model. It comes within one point of the best BFCL result at 76%, ranks third on MATH-500 at 96%, and ranks fourth on GPQA Diamond at 67%.
• Qwen3.5 9B (Non-reasoning) is the strongest tool caller, with 77% on BFCL, and the strongest scientific reasoner, with 79% on GPQA Diamond.
• Ornith-1.0-9B records the highest factual recall on AA-Omniscience, with 15% accuracy.
• LFM2.5-2.6B achieves the highest IFBench score among the models tested on the iPhone, at 59%. It also exceeds 90% on MATH-500 and shows a 79% non-hallucination rate. Nanbeige4.2-3B records 33%, while the two Qwen3.5 9B variants record 24% and 1%.

The non-hallucination result raises LFM2.5-2.6B's overall average to the same level as Nanbeige4.2-3B. Considering only the other four evaluations, Nanbeige4.2-3B, Ornith-1.0-9B, and Qwen3.5 9B (Reasoning) would all rank above it.

Generation Time and Memory

End-to-end generation time varies by a factor of 30, from 0.9 seconds for LFM2.5-230M to 26.7 seconds for Falcon-H1R-7B. These measurements process the same 1,024-token prompt and generate a 256-token response.

Peak memory use during testing at 4K context ranges from 0.4 GB for the smallest models to 6.9 GB for Ornith-1.0-9B and Qwen3.5 9B. On a 12 GB phone, the upper end leaves limited memory for the operating system and other applications.

How the Results Are Measured

Inference results in this launch were measured on an iPhone 17 Pro. Results for additional devices are reported separately and can change as models and hardware are added. Intelligence results are device-agnostic because every model is evaluated using the same quantized build that a device would load.

The benchmark reports several views of the data:

• Average intelligence score against end-to-end generation time
• Average intelligence score against peak memory
• Generation time for processing a 1,024-token prompt and producing a 256-token response
• Output speed and peak memory
• Average score with 16K and 64K maximum contexts
• Average score with a one-minute maximum answer time
• Context-budget overruns, meaning generations that stop at the 16K limit before completing
• Individual scores for tool calling, instruction following, knowledge accuracy, non-hallucination, scientific reasoning, and quantitative reasoning

End-to-end generation time is the total wall-clock time required to process the prompt and generate the response. It reflects the hardware and model architecture but does not account for model verbosity or differences in the number of turns used during interaction.

Next Steps

The benchmarking program is planned to expand in several directions:

• Comparing models by memory footprint instead of using a single quantization, allowing heavily quantized large models and higher-precision small models to compete under equal memory constraints
• Adding models as they are released
• Supporting additional inference frameworks beyond llama.cpp and testing more devices
• Investigating evaluations designed specifically for small models and on-device use