Ai2 found a reworded prompt dropped GPT-5.4's novelty accuracy from 93.5% to 40.9%

The Allen Institute for AI (Ai2) released a benchmark on October 27 that tests whether large language models can tell a genuinely new research idea from a recycled one. Its headline result is that they mostly cannot do it reliably: changing a few words in the prompt moved GPT-5.4's accuracy on one task from 93.5% to 40.9%, worse than a coin flip.

The finding matters because AI systems that generate research ideas are often graded by another model, which Ai2 calls a novelty judge. If the judge's verdict depends on phrasing, claims that one ideation system is "more creative" than another rest on shaky ground.

How Ai2 built a test set with known answers

Instead of using paper acceptance as a stand-in for originality, the team mined ICLR 2026 peer reviews for passages where reviewers explicitly discussed it. The high-novelty pool holds 154 ideas from accepted papers in the top 10% of their research area, where reviewers praised the originality and none disputed it. The low-novelty pool flips those rules: 145 ideas from bottom-10% papers that were turned down, where reviewers raised originality concerns and none praised it.

A second set paired the same 154 human ideas with 154 ideas from a deliberately simple LLM generator that had no literature search or feedback. Ai2 treats those generated ideas as less novel, and says plainly that these are weak labels. An expert who checked 30 of the pairs preferred the human idea every time.

Six judges were tested: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6.

The prompt that broke the judge

The 93.5% score came from a prompt that told GPT-5.4 reviewers had judged one idea per pair as the novel one. A plainer prompt that simply asked which idea was novel sent accuracy down to 40.9%, and the model flipped its answer on more than half of the identical pairs.

On the human-only pairs, the same wording change moved GPT-5.4 by only about 7 points, which Ai2 reads as a warning: a judge can look stable on human research and fall apart on the machine-written ideas it is actually used for. Human ideas were not safe either, since the change moved Claude Opus 4.6 by 21 points.

When the judges compared two AI generators against each other, between a quarter and a third of the verdicts flipped on average as the prompts changed. No single prompt worked best for every model.

More search did not buy better answers

GPT-5.6 Sol, run as an agent that searched arXiv on its own with the top reasoning setting, scored 0.78 macro-F1, about 11 points below Claude Opus 4.6 with five pre-retrieved abstracts, while costing roughly 10 times more per decision.

Dedicated tools fared no better. A plain Claude Opus 4.6 prompt with low reasoning and a majority vote over three calls scored 0.854 at a total cost of $2.59. Idea Novelty Checker scored 0.642 for $30.66, and the novelty evaluator in Sakana AI's AI Scientist scored 0.772 for $82.05, both running on Claude Opus 4.6.

"Our results suggest that LLM-based novelty evaluation remains a major open problem," Ai2 wrote. The benchmark covers only ICLR machine learning reviews, so how judges behave on biology or physics ideas is still untested. The data and code are public for others to try.