Anthropic Releases New Tool to Measure Political Bias in AI

Anthropic has unveiled a new open-source evaluation framework designed to measure political bias in AI models, with a focus on ensuring Claude remains politically even-handed. The company says the goal is to help AI systems handle political topics with fairness, accuracy, and respect for diverse viewpoints.

Why Anthropic Built This

According to Anthropic, people want AI to engage in political conversations without pushing any ideology or favoring one side. Claude is trained to avoid unsolicited political opinions, use neutral language, and clearly present multiple perspectives. The model is also designed to avoid influencing users on sensitive issues such as elections, policies, and social debates.

To support this behavior, Anthropic uses a mix of system prompts and “character traits” that encourage balanced analysis and non-partisan communication.

How the New Evaluation Works

The company’s new method uses Paired Prompts — two prompts on the same political issue but written from opposing viewpoints. Each response is graded on:

• Even-handedness: showing similar depth and quality for both sides

• Opposing perspectives: acknowledging counterarguments

• Refusal rate: avoiding unnecessary declines to answer

This automated evaluation covers 1,350 prompt pairs, 150 political topics, and 9 task types, ranging from essays to stories to direct opinion questions.

How Leading Models Performed

Anthropic tested its models against several major AI systems. Results showed:

• Claude Sonnet 4.5 and Claude Opus 4.1 were among the most even-handed

• Gemini 2.5 Pro and Grok 4 performed at similar high levels

• GPT-5 showed moderate performance

• Llama 4 had the highest imbalance and refusal rates

Anthropic also validated the scoring using multiple graders, including GPT-5, which showed strong consistency.

Open-Source Contribution

Anthropic has open-sourced the dataset, grader prompts, and full methodology. The company hopes other developers will use and improve the framework, helping the AI industry move toward shared standards for political fairness.