Artificial Analysis Introduces a Search API Benchmark for Agent-Based Search

Artificial Analysis Introduces a Search API Benchmark for Agent-Based Search

Artificial Analysis has introduced the Artificial Analysis Search Index, a benchmark for evaluating search API providers on quality, cost, and speed when used by an AI agent.

The benchmark compares providers across three search-focused evaluations. Each provider is tested with the same candidate model, harness, tasks, and settings. The results are also compared with a model-only baseline that uses the same model without search tools.

The launch results, dated August 18, 2026, include 11 provider configurations from seven search providers.

Artificial Analysis Search Index

The Search Index is the equal-weighted mean of three benchmark scores:

• DeepSearchQA F1
• BrowseComp accuracy
• AA-Omniscience accuracy

Scores are reported on a 0 to 100 scale, with higher values indicating better benchmark performance. A strong result combines a high Search Index score with low latency and cost.

The baseline uses GPT-5.6 Luna (medium) on the same tasks in a single-shot configuration without search tools. This provides a reference for measuring how much each search provider improves the model's performance beyond its internal knowledge.

Every provider result uses the same candidate model, harness, and settings. The search API provider is the only variable.

How the benchmark works

Each evaluation pairs one search API provider with GPT-5.6 Luna (medium). The model runs inside Stirrup, an open-source agent harness, with two tools:

• web_search
• web_fetch

The web_search tool sends the model's query to the selected provider and returns its native response payload. Content modes are standardized to snippets, the maximum result count is 10, and known contamination sources are filtered out.

The model can use up to 25 turns for each task. When it has enough information, it calls finish and submits an answer. A benchmark-specific grader then evaluates the response. If the model uses all 25 turns without calling finish, no answer is submitted and the task receives a score of zero.

The benchmark keeps the following constants fixed:

• Candidate model: GPT-5.6 Luna (medium)
• Harness: Stirrup
• Tools: web_search, web_fetch, and finish
• Maximum turns: 25
• Tasks and graders: identical across providers

Benchmark components

DeepSearchQA

DeepSearchQA contains broad research questions that require multiple searches. The expected answers are lists of items, and an LLM grader evaluates each response using F1 over the answer items. The full evaluation split contains 900 tasks.

A representative question asks for the names of spells in Old School Runescape that teleport into the wilderness and require more than four runes.

Answer: Dareeyak Teleport, Ghorrock Teleport

BrowseComp

BrowseComp evaluates difficult-to-find facts that require multi-hop browsing. The benchmark uses a hard 200-sample subset from the full evaluation pool, and the grader checks exact-answer accuracy.

A representative task describes a structure in Eastern Australia that can be visited on foot, was rebuilt in 2016, is longer than 50 meters, can be seen from another similar structure, hosts an annual dinner, and was originally built for a different purpose.

Answer: Shorncliffe Pier, Brisbane, Australia

AA-Omniscience

AA-Omniscience is a private set of 600 factual questions balanced across six domains. The grader measures accuracy. Its score is intended to show how much search contributes beyond the model's internal knowledge.

A representative question asks where the three-piece jazz band in Karen Russell's short story "St. Lucy's Home for Girls Raised by Wolves" was hired for the Debutante Ball.

Answer: West Toowoomba

These examples describe the types of problems in each benchmark and are not benchmark samples.

Cost per task

Total cost per task combines:

1. Search API charges for the provider's calls, using the provider's listed pricing.
2. Candidate model costs for all input, cached, reasoning, and output tokens used during the task.

Search results can affect model cost. Focused results may reduce token usage, while lower-quality results can cause the model to perform more searches or use more reasoning tokens.

For example, Parallel Search (advanced) has a higher search cost per task than Parallel Search (basic), at $0.048 compared with $0.045. Its total cost per task is lower, however, at $0.084 compared with $0.11.

The advanced tier returns higher-quality results, and the model uses approximately 169,000 tokens per task instead of approximately 339,000. As a result, model cost falls from $0.07 to $0.036. The tradeoff is higher task time, 35.9 seconds compared with 20.4 seconds.

Firecrawl Search records a Search Index score of 73 at a total cost of $0.075 per task. Its result is helped by credit-based pricing, with search costs of approximately $0.03 per task, and by the quality of its returned results. It also has the longest average task time in the launch results, at 55.0 seconds.

Exa Search (fast) completes individual search queries faster than Exa Search (auto), at 0.91 seconds compared with 2.10 seconds. It costs more per task, though, at $0.16 compared with $0.13. The fast tier performs more searches per task, 11.2 compared with 9.37, and the model uses more tokens, approximately 392,000 compared with approximately 299,000. Both search and model costs therefore increase.

Time per task

Time per task combines measured search time with model time. Model time is derived by dividing answer and reasoning tokens per task by the model's canonical answer output speed. Search time is the measured time spent in web_search calls.

A provider can be fast for each individual search call while still increasing total task time if the model needs to search more often or produce more tokens.

Keenable Search (realtime) has the fastest average search calls in the launch data, at 0.34 seconds per query. It also has the shortest total task time, 15.1 seconds, and a Search Index score of 67.

Among Parallel's tiers, Parallel Search (turbo) has the fastest average search calls, at 0.51 seconds per query compared with 1.03 seconds for Parallel Search (basic). The basic tier scores higher on quality, however, at 73 compared with 67. Turbo also performs more searches per task, 13.6 compared with 9.03, leaving the tiers relatively close in total task time: 18.8 seconds compared with 20.4 seconds.

The model-only baseline takes approximately 13.8 seconds per task without search time. DeepSearchQA and BrowseComp help explain why search-based configurations take longer. These tasks benefit from combining a model with search results that supply additional context.

For BrowseComp, the model-only baseline produces approximately 4,100 output tokens per task without search, compared with approximately 1,600 to 2,900 tokens for provider configurations.

AA-Omniscience shows the opposite pattern. With search tools, the model reads and reconciles sources before responding, producing roughly two to three times as many output tokens as the baseline.

Launch leaderboard

The search and model cost figures represent the average cost of 1,000 benchmark tasks. The model-only baseline performs no searches, so it has no search cost.

Interpreting the results

The launch data shows that search provider performance involves tradeoffs among answer quality, search cost, model-token cost, and latency.

Search quality can reduce overall model usage by returning more focused information. Conversely, a provider with a low per-query price or latency may still produce a more expensive or slower task if the model must issue additional searches or process larger responses.

The model-only baseline provides a reference point for measuring search-related improvements. It scores 33 on the Artificial Analysis Search Index, compared with scores ranging from 65 to 75 among the listed provider configurations.

The methodology defines the full workflow, metrics, fixed settings, and grading process.