ALTK-Evolve Uses Learned Memory to Lead τ²-bench Customer-Service Results

ALTK-Evolve Uses Learned Memory to Lead τ²-bench Customer-Service Results

ALTK-Evolve learns and applies a task-specific playbook with GPT-4.1. On τ²-bench customer-service tasks, the approach improves reliability without changing the model weights or using a frontier model.

Learned memory improves a basic GPT-4.1 agent

Using GPT-4.1, a non-reasoning model, a basic customer-service agent achieves 67.5% Pass^4 on τ²-bench retail. This means it solves every one of four independent attempts for 67.5% of the tasks. In the October 12, 2026 leaderboard snapshot, that result is higher than every other retail entry.

The agent uses τ²-bench's standard ReAct scaffold, with learned guidelines appended to its system prompt.

ALTK-Evolve creates the playbook with GPT-4.1. It examines the agent's attempts on training tasks and distills successful and unsuccessful behaviors into concise rules, such as how to authenticate a customer, confirm a transaction, or process a return. The rules are added to the prompt for later conversations. The same model performs guideline extraction and serves as the agent, with no weight updates or human annotation.

Earlier work applied this method to coding agents. This evaluation examines customer service, where success depends on following domain policies, interpreting customer requests, and executing the correct tool actions.

τ²-bench evaluates policy-bound customer service

τ²-bench, developed by Sierra, evaluates customer-service conversations under operational constraints. Each task combines:

• An LLM-simulated customer with a specific goal
• Domain tools, such as order lookup, refunds, and flight rebooking
• A written policy that the agent must follow

The agent must gather the required information, stay within policy, and perform the correct actions, some of which may be irreversible. A task is successful only when the final database state matches the expected result, producing a binary reward. The evaluation uses τ²-bench's retail and airline domains.

The experiments use τ²-bench's standard basic ReAct agent with a two-phase THINK → ACT loop. The agent uses gpt-4.1, the customer simulator uses gpt-5.5, and each task is run four times. The only modification is the addition of learned guidelines to the system prompt. The policy, tools, and agent scaffold remain unchanged.

The guidelines are appended after the domain policy as a Markdown list. Examples include:

## Guidelines learned from past task attempts (follow these carefully):
- Authenticate the user before taking any order-related action. If email is unavailable, verify identity via full name and zip code using the find_user_id_by_name_zip API.
- Confirm intended changes, such as modifications, cancellations, returns, and exchanges, with a structured summary. Require explicit approval before executing database updates.
- For returns and exchanges, explain the refund with an itemized breakdown and clarify the expected timing.
- Take one action per step. Do not combine a tool call with a user-facing message, and wait for the tool result before replying.
- Clean and parse names before searching. After several failed lookups, state the exact format required.

Guidelines are extracted from the agent's trajectories on τ²-bench training tasks using gpt-4.1, then applied unchanged to the held-out test set.

Guideline quantity affects capability and reliability

Each guideline has a support count, representing the number of distinct training tasks from which it was extracted. A higher count indicates that the rule generalized across more tasks.

Three configurations were evaluated:

Each task is attempted four times, and results are reported in two ways:

• Mean Accuracy: The average success rate across four trials, measuring capability.
• Pass^4: The percentage of tasks that succeed in all four trials, measuring reliability.

Pass^4 is particularly relevant to customer support. An agent that resolves a refund correctly three times out of four still fails a quarter of customers. Pass^4 measures whether the result is consistent across every attempt.

Held-out test results

Mean Accuracy increases in both domains: retail rises by 4.4 percentage points, from 78.1% to 82.5%, while airline rises by 7.5 points, from 57.5% to 65.0%.

Reliability also improves. Retail Pass^4 increases by 10 points, from 57.5% to 67.5%, and airline Pass^4 rises by 10 points, from 35.0% to 45.0%.

The best amount of guidance varies by domain

The two domains respond differently to guideline quantity. In retail, the larger lossless set produces the strongest results across both metrics. The smaller sup3 set provides only a limited improvement over baseline.

In airline, lossless produces the highest accuracy, at 65%, but the compact sup3 set produces the highest Pass^4, at 45%, compared with 40% for lossless. This indicates that support thresholds may need to be tuned for each domain rather than fixed universally.

Improvements require more input tokens, not more turns

Guidelines increase prompt input tokens, but they do not make conversations longer.

To compare conversation length, the evaluation examines matched runs that were solved under both baseline and guideline settings. These runs use the same scenario and seed.

On the matched sets, guideline-based conversations are slightly shorter. Lossless reduces average turns by approximately 0.3 to 0.4 in both domains. No configuration increases conversation length.

The additional cost comes from the larger system prompt sent with each model call. On the full base split, retail cost per conversation increases from $0.084 for baseline to $0.139 for lossless, a 65% increase. Airline cost rises from $0.081 to $0.107, a 32% increase.

The sup3 configuration costs $0.095 per retail conversation, or 12% above baseline, and $0.097 per airline conversation, or 19% above baseline.

Because fewer turns mean fewer sequential model round trips, the guideline configurations also reduce end-to-end completion time on the matched runs. The guideline block is identical across calls, so prompt caching can reduce the repeated input cost.

Learned guidelines also help frontier models

The initial experiments use the older, non-reasoning gpt-4.1. To test whether learned guidelines also help a stronger model, the airline evaluation was repeated with gpt-5.5 for both the agent and guideline extraction.

The same τ²-bench ReAct scaffold was used, and guidelines were mined from the agent's own runs. In this setup, no guideline reached support 3. The high-support configuration therefore uses sup2, containing guidelines with support of at least 2.

On the same 20 held-out airline tasks, lossless guidelines increase accuracy by 10 points, from 72.5% to 82.5%, and Pass^4 by 15 points, from 55.0% to 70.0%.

The improvement is larger than the 10-point reliability gain observed with gpt-4.1 on these tasks. The lossless result was also reproduced consistently across reruns, while baseline reliability varied between runs.

ALTK-Evolve's leaderboard results

τ²-bench leaderboard submissions are scored on the full base split, which combines the training and test tasks, and are reported using Pass^4. These results therefore differ from the held-out test results above.

• Retail: The agent with learned guidelines reaches 67.5% Pass^4, placing it first in the retail comparison. It exceeds every frontier model and RAFT-30B-A3B, which scores 61.4% after fine-tuning on synthetic τ²-bench retail data. The prompt-based gpt-4.1 system leads RAFT-30B-A3B by 6.1 percentage points.
• Airline with gpt-5.5: The learned-guideline agent reaches 82.0% Pass^4, placing first and exceeding frontier entries that score between 66% and 72%.
• Airline with gpt-4.1: The compact sup3 configuration reaches 44% Pass^4, placing eighth. It exceeds GPT-5.2 without reasoning and nearly matches Claude Sonnet 4.5.

Across both domains, the results show that guidelines learned around a base model can improve agents using different model strengths. The approach brings an agent using the older gpt-4.1 into the range of frontier systems on retail and takes an agent using gpt-5.5 to the top of the airline comparison.

Agents can improve without weight updates

ALTK-Evolve improves a standard agent using a standard model through prompt-level guidelines extracted automatically from the agent's own training-task runs. The method requires no weight updates, human annotation, or test-set leakage.

On retail, the resulting system matches or exceeds a model fine-tuned on synthetic τ²-bench data. The experiments also show that learned memory can improve both capability and reliability, while conversations remain similar in length or become slightly shorter.

Resources

• ALTK-Evolve library
• τ²-bench
• Retail leaderboard submission, PR #542
• Airline leaderboard submission, PR #551