HealthBench and HealthBench Professional Evaluate AI Models in Realistic Health Conversations

HealthBench and HealthBench Professional Evaluate AI Models in Realistic Health Conversations

OpenAI’s HealthBench and HealthBench Professional benchmarks assess how AI models respond to realistic health-related conversations. Physician-created rubrics are used to grade model replies.

HealthBench contains 5,000 conversations involving everyday users and health workers. The conversations cover seven themes, including emergency referrals and global health. Each rubric criterion is assigned to one of five evaluation axes:

• Accuracy
• Completeness
• Communication quality
• Context awareness
• Instruction following

HealthBench Professional is a smaller benchmark of 525 conversations, all involving clinicians. It covers consultation, writing, and research tasks, including adversarial and difficult scenarios intended to challenge models.

The leaderboards report both the raw score, representing the share of rubric points earned, and OpenAI’s length-adjusted score. The adjusted score reduces results for replies longer than 2,000 characters and increases results for shorter replies. Following OpenAI’s original implementation, replies are graded by GPT-5.4 (low) for HealthBench Professional and GPT-4.1 for HealthBench.

HealthBench Professional results

• GPT-6 Astra and GPT-6.1 Sol lead the leaderboard: GPT-6 Astra (max) achieves a length-adjusted score of 66.0%, followed by GPT-6.1 Sol (max) at 65.1% and Claude Sonnet 5.5 (max with fallback) at 63.0%.

• OpenAI models occupy the entire score versus Cost per Task Pareto frontier: The frontier runs from GPT-6 Luna (max), with a score of 61.9% and a cost of $0.004 per task, through GPT-6 Sol (max) at 62.3% and $0.036, GPT-6.1 Sol (max) at 65.1% and $0.047, and GPT-6 Astra (max) at 66.0% and $0.27. GPT-6 Luna (max) scores four points below GPT-6 Astra (max) at less than 2% of the cost.

• Claude Opus 5.5 and Claude Fable 5.1 receive the largest length penalties: Both models average approximately 5,900 characters per answer, nearly three times the 2,000-character threshold, and lose more than 11 points after adjustment. Claude Opus 5.5 (max with fallback) has the highest raw score, at 71.4%, but falls to sixth place with a length-adjusted score of 59.9%. GPT-6 Sol (max) is the only model averaging fewer than 2,000 characters, at approximately 1,600, and the only model whose score increases after adjustment.

HealthBench results

• Claude Sonnet 5.5 leads HealthBench: Claude Sonnet 5.5 (max with fallback) records a length-adjusted score of 62.1%, ahead of DeepSeek V4.1 Flash (max) at 60.9% and GPT-6.1 Sol (max) at 58.5%.

• GPT-6 Luna, DeepSeek V4.1 Flash, and Claude Sonnet 5.5 form the score versus Cost per Task Pareto frontier: GPT-6 Luna (max) scores 54.6% at $0.002 per task, while DeepSeek V4.1 Flash (max) scores 60.9% at $0.005 per task. Claude Sonnet 5.5 (max with fallback) has an average Cost per Task of $0.08, approximately 16 times the cost of DeepSeek V4.1 Flash for an increase of about one percentage point.

• Shorter answers trade completeness for communication quality: Each HealthBench rubric item is assigned to one of the five evaluation axes. Completeness measures whether a reply includes all important information. Communication Quality measures whether the response is clear, well structured, and delivered at an appropriate length and level of detail. GPT-6 Sol (max) produces the shortest answers, at approximately 1,000 characters, and has the highest raw Communication Quality score at 81%. It also has the lowest Completeness score, at 34%. Claude Sonnet 5.5 (max with fallback) leads on Completeness with 63%.

The complete leaderboards are available for HealthBench Professional and HealthBench.