How to Evaluate CPU Performance for Agentic AI

Why CPUs Matter Again in Agentic AI

The shift from machine learning to chatbots and then to agentic AI has restored the CPU to a central role in computing. CPUs dominated commercial computing for six decades before GPUs and XPUs became important for highly parallel vector and matrix operations. Agentic AI has increased demand for CPU capacity because AI host CPUs and agentic AI sandboxes execute much of the work surrounding model inference.

The CPU market is expected to grow from $26 billion in 2025 to $246 billion by 2030, with much of that growth attributed to AI host CPUs and agentic AI infrastructure.

Not all CPUs provide the same capabilities. Hyperscalers, cloud builders, AI model developers, neoclouds, sovereign AI centers, and enterprises must balance fleet standardization against the need to match hardware to different parts of increasingly complex workflows. These workflows combine human-written and AI-generated code, model calls, retrieval, databases, tools, and sandboxed execution. Inference itself is also changing rapidly, making CPU selection more difficult.

At datacenter scale, the important question is how many successful tasks can be completed within a rack's power and thermal limits. Arm's proposed approach evaluates CPU performance at the socket, system, and rack levels. It considers both single-thread performance and the ability to scale cores within a socket, across a system, and throughout a rack. The objective is to understand how many AI agents can run concurrently while meeting service-level agreements.

Measuring Complete Agentic Workflows

The technology industry has previously developed benchmarks for complex, multi-tier workloads. TPC-E, for example, simulated a warehouse operation involving interactive and batch workloads across database, application, and infrastructure tiers.

“Multi-tier and stateful computing are not new, and existing cloud and enterprise benchmarks remain important for understanding individual parts of the system,” Jeff Defilippi, Senior Director, Product Management at Arm, told The Next Platform. “What changes with agentic AI is how those components are composed. A single task can dynamically chain model calls, retrieval, database transactions, code execution, tool invocations and state changes, with the result of one step determining what happens next. That makes successful task completion, not any single component metric, the more meaningful unit of performance.”

Existing benchmarks remain useful for identifying the sources of performance differences. However, they need to be connected to end-to-end measurements that show how quickly and efficiently a system completes a task, and how performance changes as more agents run concurrently and server utilization increases.

Arm is not proposing a single definitive agentic AI benchmark. Instead, it advocates a methodology that combines established component benchmarks with agentic workload measurements and end-to-end task evaluation. InferenceX and TPCx-AI can represent important parts of an AI workflow, including inference serving and machine learning pipelines. An agentic system also includes CPU-side orchestration, retrieval, state management, tool execution, sandboxing, and security, which these discrete benchmarks were not designed to measure.

“The goal is therefore to take a holistic view of the agentic flow while retaining component-level measurements that let engineers understand where performance differences originate,” Defilippi said. “Arm intends to characterize a broad set of these behaviors, from targeted component tests through higher-level metrics such as wall-clock task completion, agent density, and efficiency. The methodology is applicable to AGI CPU systems, but we are defining the workload and measurement framework rather than announcing a specific AGI CPU cluster benchmark.”

CPUs, Accelerators, Memory, and Networking

The methodology applies beyond CPUs because agentic AI workloads also use GPUs and other XPU accelerators. Evaluations should measure CPUs, accelerators, memory subsystems, and networking, while also examining orchestration, retrieval, and tool deployment and execution.

A useful testing methodology needs two complementary layers:

• Controlled component tests that expose individual bottlenecks
• End-to-end tests that measure user-facing outcomes

Sequential latency is especially important in agent loops. CPUs are moderately parallel processors with cache hierarchies and clock speeds designed for strong sequential performance. They often coordinate with GPUs or combinations of GPUs and XPUs that perform prefill token processing and decode token generation. The CPU then helps drive subsequent queries, tool activation, and code generation.

“Single-threaded performance matters because many parts of an agent loop are latency-sensitive and sit on the critical path,” Defilippi said. “But it is only one dimension. Agent infrastructure also has to support many agents at once, so core count, memory bandwidth, cache capacity, scheduling efficiency and sustained performance under high utilization become increasingly important as concurrency rises.”

Latency and Concurrency

Benchmarking becomes more difficult when one logical operation spans multiple dependent steps. An agent may reason about a task, invoke a tool, wait for a database or API response, update its state, return to model inference, and decide what to do next. Because later steps often depend on earlier results, latency accumulates along the critical path even when parts of the system operate in parallel.

“Many agentic workflows therefore have both a latency problem and a concurrency problem,” Defilippi explained. “For an agent, the next action often cannot begin until the preceding dependency completes, so CPU-side execution, memory access, model inference, tool calls and network latency can all contribute to task-completion time. Reducing latency in the CPU-visible portions of that path can move the agent to its next action sooner and, where accelerators are involved, return work to the GPU or XPU sooner as well.”

At the same time, infrastructure rarely runs just one agent. Other agents and independent workflow components can execute while one dependency is waiting. As concurrency increases, core count, memory bandwidth, cache capacity, scheduling efficiency, and sustained performance become more important. The system must balance responsive execution along each dependency chain with enough aggregate throughput to support many agents simultaneously.

This leads to three practical evaluation levels:

1. Core level: Measure latency and per-thread performance for work on the critical path.
2. System level: Measure throughput, concurrency, memory behavior, and sustained performance under high utilization.
3. Rack level: Measure how much completed work the infrastructure can sustain within its power, cooling, and physical constraints.

Component measurements explain why platforms behave differently. End-to-end agentic measurements show whether those differences improve successful task completion.

CPU Effects on Accelerator Utilization

When accelerator inference is part of an agent loop, CPU performance can affect accelerator utilization. CPUs handle request scheduling, orchestration, data preparation, runtime services, and other functions surrounding model execution. Delays on the CPU side can extend the task path, while efficient CPU execution can return work to an accelerator sooner.

Infrastructure teams should therefore evaluate systems under realistic load and concurrency instead of relying only on peak results from lightly loaded servers.

System architecture must reflect the workload. The appropriate choices for compute engines, memory, networking, and storage depend on how those resources are used. Predicting how agentic AI workloads will evolve over the next four to five years is difficult, but the tradeoff between latency and concurrency is expected to remain important over the useful life of systems installed today.

“Where inference serving is part of the agent loop, CPU-visible functions such as scheduling, tokenization, serialization and KV-cache management can put pressure on the memory subsystem,” Defilippi said. “That makes memory bandwidth, bandwidth per core, cache behavior and memory locality relevant to how efficiently the system advances.”

Waiting for a tool response creates a different pattern. The CPU can schedule other work while an agent waits for a database, API, or sandboxed application. At low concurrency, the dependency has a major effect on an individual task's completion time. At higher concurrency, the ability to run other agents makes throughput, memory bandwidth, and scheduling efficiency increasingly important.

Rack-Level Economics

At scale, performance measurements become questions about rack economics. Datacenters operate within fixed power, cooling, and physical constraints. The relevant measure is not simply how much performance one processor can deliver, but how much useful agent work the infrastructure can sustain within those limits.

Metrics such as tasks per rack and energy per task can complement CPU-level performance. Completing more tasks within the same rack-level power envelope can improve infrastructure efficiency and reduce task cost. Thermal headroom can also help systems absorb activity bursts without compromising performance.

These comparisons should account for scheduled batches because some agentic workloads are batch-oriented. More broadly, persistent and highly utilized agent services make sustained efficiency, density, and throughput per rack important regardless of whether individual tasks are interactive or batch workloads.

Arm AGI CPU Testing

Arm has applied this framework to its AGI CPU. Using workloads aligned with the proposed agentic methodology, Arm authorized an independent third party to evaluate AGI CPU performance against AMD EPYC Turin across CPU-visible portions of agentic workflows, including tooling, orchestration, reasoning, and planning.

The testing used measured server-level performance as the basis for rack-level modeling. The results reported approximately 2X the performance per rack compared with AMD EPYC 9755. The comparison illustrates why rack-level performance can produce a different conclusion from an isolated core or socket benchmark.

Evaluating capacity for agentic AI is therefore a complex planning task. Organizations need measurements that connect component behavior with complete workflow outcomes, from individual CPU threads to systems and racks. Component vendors and system manufacturers will play an important role in developing the data needed to make those decisions.