VIALS-AA: Different Paths to Scientific Image Reasoning

VIALS-AA: Different Paths to Scientific Image Reasoning

VIALS-AA is an implementation of Handshake AI's VIALS benchmark. It evaluates whether language models can interpret scientific images and answer questions using the evidence shown in them. The results indicate that models can reach correct answers through very different processes, and that extensive image inspection does not always prevent mistakes.

This launch analysis reports results as of November 6, 2026. VIALS-AA contains 161 tasks, and each task is run three times with Stirrup, an open-source reference harness that provides code execution and image-inspection tools.

Key findings

• Gemini 4 Carbon (high) leads with a score of 64%. It answers all three repeats correctly on 103 of 161 tasks, two more than GPT-6 Astra (max). Gemini 4 Carbon (high), GPT-6 Luna (max), GPT-6.1 Sol (max), GPT-6.1 Terra (max), and GPT-6 Astra (max) form the score-cost Pareto frontier among the tested models.
• More turns and visual inspection do not always improve results. Claude Sonnet 5.5 (max, with fallback) and GPT-6.1 Sol (max) score similarly, at 60% and 59%, respectively. However, Claude Sonnet 5.5 makes 3.6 times as many tool calls per task.
• Consistency distinguishes models with similar pass@3 rates. Kimi K3.1 Pro (max) and GPT-6 Astra (max) each solve about three-quarters of tasks at least once. Requiring all three repeats to be correct produces scores of 45% for Kimi K3.1 Pro (max) and 63% for GPT-6 Astra (max). VIALS-AA uses this stricter measure because an answer that changes between runs cannot be considered reliable for scientific work.
• Cost per task varies by nearly 99 times. The range extends from $0.02 for GPT-6 Luna (max) to $1.77 for Grok 4.7 (xhigh). GPT-6.1 Sol (max) scores 59% at $0.06 per task, while Claude Opus 5.5 (max, with fallback) scores 61% at about 30 times the cost.

How the evaluation measures cost

Cost is calculated from input, cache-read, cache-write, reasoning, and answer-token prices, divided by the number of tasks. Token usage is averaged across the three independent repeats for each task, using each model's representative cache-hit rate. The calculation includes contestant model calls but excludes judge calls.

Example trajectories through scientific images

The first example asks models to locate protein spot 22 in a gel electrophoresis image and estimate its molecular-weight range using the vertical axis. The models use different approaches, including brief visual inspection, explicit measurement, and repeated checks with annotations and processed views.

Short visual inspection

GPT-6 Astra (max) checks the image dimensions, opens the original, creates a 4× enlargement, and inspects it. It then gives the correct answer in six turns. The model does not measure pixels, adjust contrast, or draw reference lines. Instead, it relies on its visual interpretation.

The answer is 40-50 kDa, matching the reference.

Explicit measurement and calculation

Claude Sonnet 5.5 (max) enlarges the gel and crops the region around spot 22. It measures ladder bands, inspects pixels near the arrow, draws reference lines, enhances contrast, and interpolates molecular weight. These measurements place the spot in the 40-50 kDa range.

The model finishes in 18 turns. Its interpolation produces the following estimates:

22 number label centre     y= 159.0  MW(log-linear)= 53.0 kDa   MW(pchip)= 52.7 kDa
arrowhead tip              y= 168.5  MW(log-linear)= 49.7 kDa   MW(pchip)= 49.7 kDa
faint spot centre          y= 172.0  MW(log-linear)= 48.7 kDa   MW(pchip)= 48.6 kDa
spot lower extent          y= 174.0  MW(log-linear)= 48.1 kDa   MW(pchip)= 48.0 kDa

It also answers 40-50 kDa, matching the reference.

Repeated checks that produce a visual error

Grok 4.7 (xhigh) repeatedly examines the region using grids, thresholds, reference lines, and labeled pixels. It uses 27 turns and 20 image views, but answers 50-70 kDa.

In its final annotated view, Grok circles the printed “22” in orange and the arrow tip in red, placing the tip on its green 50 kDa guide. Its explanation relies on the label and leader rather than the faint protein spot. The interpolation performed by Claude Sonnet 5.5 separates the label, at about 53 kDa, from the spot, at about 48.7 kDa, supporting the reference range of 40-50 kDa.

Interpreting images without native vision

Vision-enabled models are a natural fit for VIALS-AA because they can inspect a scientific figure, read labels, and connect visual features to the question. The evaluation also tests how a model without native vision approaches the same tasks.

GLM-5.3 (max) receives the same image files and code-execution access. Without native vision, it extracts evidence through code. In the protein-spot-22 example, it loads the image into arrays, enlarges and thresholds regions for OCR, compares pixel intensities, and prints coordinates and text maps. It uses those outputs to locate labels, measure the molecular-weight scale, and select a candidate spot.

The comparison illustrates both the additional work involved and its limitations. GLM-5.3-Flash (max) combines code with ten image views, finishes in 30 turns, and gives the correct range. GLM-5.3 (max) takes 54 turns and makes 57 code calls without producing an image view, then answers incorrectly.

Tool usage differs substantially between models

Tool calls were classified across all 19 models in the original test set, covering 9,177 rollouts. Most calls concerned visual evidence:

• 38.3% of tool calls viewed images.
• 47.3% prepared or measured images, including cropping, enhancement, and pixel-based measurement.

Image measurement represented only 5-6% of tool calls for GPT-6 Astra (max) and GPT-6.1 Sol (max), compared with 25-27% for Claude Opus 5.5 (max) and Claude Sonnet 5.5 (max).

Tool-call counts are averaged across three independent repeats. Calls that finish or abandon a task are excluded. Per-task values use up to 483 observations per model. Tool calls per turn are calculated by dividing counted tool calls by completed model turns. Repeats without a recorded turn count, such as immediate safety blocks, are excluded.

Tool-call categories

Each tool call receives one primary category based on its main activity:

Image viewing comes from the view_image tool. The other groups classify code_exec calls. OCR accounts for 12.4% of calls in the Other category, and Gemini 3.8 Flash (high) accounts for 21% of OCR calls.

Not every code execution performs analysis

Gemini 3.8 Flash (high) accounts for 86.7% of the cohort's calls classified as commentary/no-op. These calls represent 5.6% of its tool usage, or approximately 3.8 of its 67.5 calls per task.

In the protein-spot-22 example, a commentary/no-op call prints an interpretation of the spot labels. The code-execution tool functions as a notepad: the model writes its interpretation, and the text is returned in the next model request.

Higher-scoring models generally use fewer tools

Models with higher Artificial Analysis Intelligence Index scores generally make fewer tool calls on VIALS-AA. However, similar VIALS-AA scores can result from very different inspection strategies. Opus and Sonnet use substantially more tool calls and image measurements than Astra and Sol.

Consistency separates the leading models

Leading open-weight models solve approximately as many tasks at least once as closed frontier models, but they are less consistent.

Kimi K3.1 Pro (max), MiMo-V2.6-Pro, and MiniMax-M3.1-Flash (max) each solve 114-119 tasks at least once. The seven top-scoring models, all closed models, solve 111-118 tasks at least once. However, the open-weight models pass all three repeats on only 48-73 tasks, compared with 95-103 for the top-scoring closed models.

The headline metric, pass^3, is the share of tasks answered correctly in all three repeats. It is used because inconsistent answers in scientific work can lead to incorrect conclusions and costly follow-up experiments.

Domain-level results

Gemini 4 Carbon (high) leads or shares the lead in three of the eight domain groups: plasmid maps, phylogenetics, and flow cytometry. It shares the flow-cytometry lead with Gemini 4 Argon (high).

The other domain leaders are:

• Claude Opus 5.5 (max), blotting
• Claude Sonnet 5.5 (max), cell counting
• GPT-6.1 Terra (max), protein structure and binding
• GPT-6 Astra (max) and GPT-6.1 Sol (max), molecular structure

Cost and token usage

Cost per task ranges from $0.02 for GPT-6 Luna (max) to $1.77 for Grok 4.7 (xhigh), a 99× spread. GPT-6.1 Sol (max) and Claude Sonnet 5.5 (max) score 59% and 60%, respectively, but Sonnet costs about 18 times as much per task.

Output tokens per task range from approximately 2.21k for GPT-6 Astra (max) to 94.6k for MiMo-V2.6-Pro, a 43× spread. Reasoning accounts for most of each model's output. Claude Sonnet 5.5 (max) uses 63.5k output tokens, including 54.6k reasoning tokens, or about 25 times as many output tokens as GPT-6.1 Sol (max) for a similar score.

Resources

• The VIALS-AA evaluation page contains the leaderboard.
• The methodology covers the implementation, grading, and scoring.
• Handshake AI's VIALS benchmark page describes the dataset and original evaluation.
• The public dataset is available on Hugging Face.
• The Stirrup harness is available on GitHub.