AA-PersonalityBench Measures How AI Models Communicate

AA-PersonalityBench Measures How AI Models Communicate

AA-PersonalityBench evaluates the personality and communication styles of AI models. A panel of large language model judges compares answers to the same question pairwise across 30 personality traits. These traits are grouped into eight personality indexes:

• Expressiveness
• Organization
• Informativeness
• Transparency
• Warmth
• Conviction
• Imagination
• Slop

The benchmark also measures 21 communication characteristics programmatically, including response length, structure, punctuation, and emphasis.

Personality indexes

Each index is calculated from the mean of the relevant traits' Elo z-scores. Results are then ranked by percentile from 1 to 100 across all models evaluated in the benchmark.

The benchmark includes models such as Claude Opus 5.5, Claude Fable 5.1, GPT-6 Astra, Gemini 4 Argon, Muse Wonder, GPT-6.1 Sol, Grok 4.7, MiMo-V2.6-Pro, and GLM-5.3.

The indexes describe how strongly a model exhibits a communication style. They are not presented as measures of whether that style is inherently desirable. For example, a model can be more or less expressive, direct, humorous, or opinionated depending on its default behavior.

Example questions and responses

Five public questions illustrate the benchmark:

1. Humorous
2. Truthful
3. Structured
4. Sycophantic
5. Eloquent

One example asks: "What's the weirdest social norm we all just accept?"

Claude Opus 5.5 answers by focusing on how people behave around strangers in crowded places, describing packed trains and elevators as situations in which people silently agree to ignore one another. The response refers to Erving Goffman's concept of "civil inattention" and gives additional examples, including birthday candles, the conventional response to "How are you?", lawns, salary secrecy, and the tooth fairy.

The displayed grading applies to one question. Trait scores are calculated from all 30 questions associated with each trait.

How the benchmark works

Two models can provide equally correct answers while producing noticeably different experiences. AA-PersonalityBench is designed to measure these differences by examining how models respond by default, without a system prompt.

1. Questions and model responses

There are 30 questions for each of the 30 traits, producing 900 questions in total. Questions are sent as single user messages without system prompts.

The evaluated traits include:

• Expressiveness: Lexically Varied, Eloquent, Literate, Natural Sounding, Metaphorical
• Organization: Structured, Scannable, Bottom-Line Summary, Answer First
• Informativeness: Information Dense, Evidence-Based, Resource-Specific, Actionable Guidance, Pragmatic
• Transparency: Truthful, Admits Uncertainty, Disclaims Assumptions, Flags Nuances
• Warmth: Friendly, Humorous, Sincere, Emotionally Intelligent
• Conviction: Direct, Confident, Opinionated, Constructive Pushback
• Imagination: Creative, Open-Ended
• Slop: Slop and Information Dense

Some questions are designed to gently encourage a particular trait, while others are more neutral. During question development, tie rates were examined to ensure that questions produced measurable differences between models.

The complete question set remains private to reduce benchmark contamination, while two sample questions per trait are published. Both judged traits and programmatically measured communication characteristics are evaluated using responses to these questions.

2. Pairwise judging

A panel of LLM judges compares model responses in pairs. Each comparison follows this process:

1. Two responses to the same question are shown in a blind comparison.
2. One judge from the panel evaluates the pair against the selected trait.
3. The judge returns one of three verdicts: response A, response B, or tie.

The judge sees the trait, its definition, the question, and the two answers, but not the model names. Reasoning text is not shown to the judges, only the model outputs.

The response order is randomized to reduce position bias. Comparisons are distributed across the panel so that each model's evaluations are balanced among judges.

3. Trait scores

Trait results use Elo ratings, reported as percentile ranks. Every model begins at the same level as an anchor model, with an Elo score of 1000.

A result includes:

• Elo: How strongly the model exhibits the trait
• 95% confidence interval: Calculated by refitting resamples of the verdicts
• Percentile: The model's rank from 1, representing the least, to 100, representing the most, across evaluated models

For example, the benchmark displays Friendly trait scores for Claude Opus 5.5, Gemini 4 Argon, Claude Fable 5.1, Claude Sonnet 5.5, and GPT-6 Astra, together with their Elo values and confidence intervals.

4. Combining traits into indexes

The 30 traits are combined into eight personality indexes by averaging their z-scores. Sycophantic is inverted when calculating the Transparency index.

For the Warmth index, the component traits are Friendly, Humorous, Sincere, and Emotionally Intelligent. Their trait scores are converted to z-scores, averaged, and mapped to an index from 1 to 100.

Z-scores and the index scale are computed across all models, not only the models shown in an individual chart.

5. Communication characteristics

The benchmark separately measures 21 communication characteristics directly from model responses. These measurements are not judged by the LLM panel.

They cover four groups:

• Length: Characters, words, sentences, paragraphs, and average sentence length
• Structure: Headings, lists, and tables
• Punctuation: Colons, contractions, exclamations, questions, parentheses, em dashes, quotes, and ellipses
• Emphasis: Bold text, italics, emojis, acronyms, and whether the response ends with a question

An illustrative response about the fall of Rome contains 232 characters, 37 words, three sentences, and two paragraphs, with an average sentence length of 7.3 words. It includes a heading and a list, and uses emphasis and punctuation that can be counted programmatically.

Across all 900 questions, the benchmark reports averages for each model. Examples include:

Personality traits

AA-PersonalityBench reports percentile ranks for each of the 30 judged traits across all evaluated models. The results can be viewed by personality index, including Expressiveness, Organization, Informativeness, Transparency, Warmth, Conviction, Imagination, and Slop.

The Expressiveness group measures qualities such as rich language, lexical variety, eloquence, literacy, natural-sounding phrasing, and metaphorical language.

Model families and clusters

Models are also grouped according to similarities across their personality traits or communication characteristics. The closer two branches meet in a clustering tree, the more alike the corresponding models are.

For personality clustering, models are compared using the correlations among their 30 trait Elo z-scores. The process starts with individual models and repeatedly joins the two groups with the smallest average distance.

One reported personality family includes:

• Grok 4.7 (xhigh)
• GPT-6 Luna (max)
• GPT-6.1 Sol (max)
• GPT-6 Astra (max)
• Claude Fable 5.1 (max with fallback)
• GLM-5.3-Flash
• GLM-5.3 (max)
• Kimi K3 (max)
• Claude Haiku 5.5 (max)
• Claude Opus 5.5 (max with fallback)
• Claude Sonnet 5.5 (max with fallback)
• Muse Spark Open (xhigh)
• Muse Wonder (max)
• Gemini 3.8 Flash (high)
• Gemini 4 Argon (high)
• K2 Horizon 375B A23B
• Mistral Large 4 Preview
• Step 5 Preview
• Qwen3.8 27B (xhigh)
• Qwen3.8 Max (0902)
• MiMo-V2.6-Pro
• Nemotron 3.5 Super VL
• Inkling (xhigh)
• Beam 1.0

Within this family, the clustering results show close groupings among GPT-6 Luna, GPT-6.1 Sol, and GPT-6 Astra; among GLM-5.3-Flash, GLM-5.3, and Kimi K3; and among Claude Haiku 5.5, Claude Opus 5.5, and Claude Sonnet 5.5. Other close groupings include Gemini 3.8 Flash with Gemini 4 Argon, K2 Horizon 375B A23B with Mistral Large 4 Preview, and Qwen3.8 27B with Qwen3.8 Max.

Comparing two models

The comparison view estimates the probability that one model's answer will be preferred over another model's answer for a selected trait. The probability is derived from the Elo gap, with 50% representing an even match.

For example, the benchmark compares GPT-6.1 Sol (max) with Nemotron 3 Ultra 550B A55B (Reasoning) on the Expressiveness index. The displayed probabilities are:

The benchmark can also produce custom personality rankings by allowing users to select and weight any of the 30 traits.

AA-PersonalityBench resources

The evaluation includes leaderboards and detailed results for every trait and personality index. Its methodology documentation covers the implementation, judge prompts, Elo fitting, and index calculations.