Can LLMs Reliably Judge Whether a Research Idea Is Novel?

Can LLMs Reliably Judge Whether a Research Idea Is Novel?

Published October 27, 2026

Ai2's study, Old Ideas, Novel Problems, examines how reliably large language models can evaluate the novelty of scientific research ideas. The release includes a benchmark, data, code, and an automated pipeline for creating additional evaluation sets.

Ai2 has developed systems intended to support several stages of scientific discovery. DataVoyager helps researchers explore and analyze structured data, AutoDiscovery generates and tests hypotheses, and Theorizer synthesizes evidence from large bodies of literature into candidate scientific theories. Together with Asta, these projects cover tasks ranging from understanding prior research to proposing, testing, and refining ideas.

Generating a plausible research idea is only part of the challenge. An idea must also contribute something genuinely new rather than repackage existing work. In many automated discovery systems, another LLM is used to make that determination. These systems are referred to as novelty judges.

A novelty judge assesses whether an idea is new relative to prior research, either independently or in comparison with another idea. There is no standard method for building one. Teams may vary the model, prompt, evaluation format, reasoning budget, and literature-retrieval system, and each choice can affect the result. In addition, novelty judges are often evaluated using human-written papers even when their intended use is to assess AI-generated ideas.

The study's results indicate that novelty evaluation is unreliable. Across six models, GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6, small changes to the evaluation setup produced substantially different judgments of the same ideas. In one experiment, changing only a few words in a prompt altered accuracy in identifying the more novel idea by more than 50 percentage points. Retrieval and additional reasoning did not consistently improve performance, and two specialized novelty-evaluation systems performed worse than a simple Claude Opus 4.6 prompt.

Building a benchmark for novelty judges

The researchers first needed examples with strong evidence about the originality of the underlying scientific contribution. They therefore built an automated pipeline around ICLR 2026 peer reviews.

Rather than treating paper acceptance as evidence of novelty, the pipeline searched reviews for passages in which reviewers explicitly discussed originality. Acceptance alone is not a reliable proxy: a paper may be accepted because of strengths in other areas, while a rejected paper may still contain an original contribution.

The researchers created a high-novelty pool of 154 human-authored research ideas. These came from accepted papers in the top-rated 10% of their ICLR research area, where reviewers positively identified novelty and no reviewer disputed it.

They also created a lower-novelty pool of 145 ideas from rejected papers in the bottom 10%. In these cases, reviewers raised concerns about originality and no reviewer positively identified the paper as novel.

In a separate review of the mining process, an annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases. None of the disagreements would have changed whether a reviewer was counted as praising or disputing a paper's novelty.

These two pools form the first evaluation setting, called Human-Only. It tests whether novelty judges can distinguish human-authored ideas with stronger or weaker evidence of novelty.

Because novelty judges are increasingly used to assess AI-generated ideas, the researchers created a second setting, Human+Generated. They paired the same 154 validated human ideas with 154 ideas produced by a simple LLM generator.

The study treated the generated ideas as less novel than their human-authored counterparts, but these are weak labels rather than verified judgments for every individual idea. The generator received only a research area and an instruction to propose a novel idea. It had no literature search, tools, iteration, or feedback. This created a conservative comparison between ideas from a basic generator and contributions from top-10% ICLR papers whose reviewers had explicitly endorsed their originality.

As an additional check, an expert independently reviewed 30 Human+Generated pairs, with many selected from cases that the strongest judge had evaluated incorrectly. The researchers also rewrote the human and AI-generated ideas into a common structure to determine whether judges were responding mainly to differences in writing style. The expert preferred the human idea in all 30 pairs and judged none of the generated ideas to be novel.

Many of the central findings do not depend on the labels. When two reasonable evaluation setups produce opposite answers for the same pair of ideas, the disagreement itself demonstrates instability.

Small changes can produce very different verdicts

The researchers tested six LLM judges, GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6, using both pointwise evaluations, which assign an absolute novelty rating, and pairwise evaluations, which select the more novel of two ideas.

They varied one factor at a time:

• The prompt
• The reasoning budget
• Whether the judge could access related papers
• The formatting of the ideas
• The number of judgments combined into a final verdict
• The model that generated an idea, for AI-generated research

Prompt wording produced the largest performance changes.

For example, GPT-5.4 evaluated the same Human+Generated pairs using two prompts. Each pair contained one validated human-authored idea and one AI-generated idea. One prompt stated that reviewers had found exactly one idea in each pair to be novel and asked the model to identify it. The other simply asked which idea was novel.

With the simpler prompt, accuracy fell from 93.5% to 40.9%, which was below chance. The model also changed its verdict on more than half of the same pairs.

On the Human-Only dataset, the same prompt change shifted accuracy by approximately seven percentage points. This suggests that a judge may appear relatively stable when evaluating human-authored research but behave much less consistently on AI-generated ideas.

Human-authored ideas were not unaffected. Claude Opus 4.6 accuracy on the Human-Only dataset changed by 21 points under the same prompt variation, although the shift was smaller than on the AI-generated ideas.

There was also no prompt that consistently worked best for every model. When ideas were evaluated individually, removing an instruction that warned judges not to treat trivial or incremental contributions as novel improved accuracy for some judges and reduced it for others. A prompt that helps one model may therefore not transfer to another.

The researchers also examined a common use of novelty judges: comparing ideas from two AI generators. They paired ideas from Claude Opus 4.5 with Claude Sonnet 4.5, and GPT-5.4 with GPT-5.1. Across the tested prompt changes, approximately one-quarter to one-third of verdicts changed on average across the six judges.

This disagreement can be measured without assuming that either generator produces more novel ideas. It shows that the judge's prompt can affect comparisons between ideation systems.

More search and reasoning did not guarantee better results

The study also tested whether additional search and reasoning improved novelty judgments. GPT-5.6 Sol received an extra-high reasoning budget and access to live, iterative search restricted to arXiv. Its results were compared with standard-retrieval judges that each received five pre-retrieved abstracts. Every judge in this comparison ran once per idea on the Human+Generated dataset.

The agentic judge achieved a macro-F1 score of 0.78, 11 points below Claude Opus 4.6 at 0.89. Its result was roughly comparable to simpler Claude Opus 4.5 and GPT-5.4 configurations. At current API prices, the agentic judge cost approximately 10 times more per decision.

Macro-F1 measures how well a judge identifies both novel and non-novel ideas, with higher values indicating better performance.

Specialized evaluators were not more accurate despite costing considerably more. On the Human+Generated set, a simple Claude Opus 4.6 prompt using low reasoning and majority voting across three calls achieved a macro-F1 of 0.854 at a total evaluation cost of $2.59.

By comparison:

• Idea Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at a cost of $30.66.
• Sakana AI's AI Scientist, whose novelty evaluator iteratively searches Semantic Scholar, scored 0.772 at a cost of $82.05.

Both dedicated evaluators used Claude Opus 4.6, with one call per decision. The simple prompted baseline was therefore more accurate and considerably less expensive than either specialized system in this evaluation.

Together, the results indicate that novelty evaluation is especially brittle on the AI-generated ideas these systems are increasingly expected to assess.

Toward more reliable evaluation of novel research

LLM-based novelty evaluation remains an open problem. Judges that appear reliable on human-authored research may behave differently on AI-generated ideas. Prompts that improve one model can reduce another model's accuracy, and additional search or computation does not necessarily produce more consistent judgments.

Future research should evaluate judges on the types of ideas they will actually assess, measure sensitivity across prompts and models, improve the use of prior literature, and create benchmarks that evolve as scientific knowledge and AI systems change.

The benchmark and reusable data pipeline provide a basis for applying the same process to new peer reviews, other scientific communities, and future generations of novelty judges. Automated ideation requires more than generating plausible research directions. It also requires dependable methods for determining whether those directions are genuinely new.