Researchers have devised a novel benchmark for artificial intelligence reasoning models using riddles from the renowned NPR Sunday Puzzle. This initiative aims to evaluate AI's problem-solving capabilities without relying on specialized or esoteric knowledge. The Sunday Puzzle, a staple of NPR programming hosted by Will Shortz, serves as the foundation for this U.S.-centric and English-only quiz game. Each week, new questions are introduced, offering a fresh challenge for both human participants and AI models alike.
The benchmark comprises approximately 600 riddles from various Sunday Puzzle episodes. Notably, the current top-performing AI model on this benchmark is o1, which achieved a score of 59%. The recently released o3-mini model, designed for high "reasoning effort," scored 47%. In contrast, DeepSeek's R1 model, which occasionally provides incorrect answers it knows to be wrong, managed to score 35%. Despite these varied performances, reasoning models such as o1 and DeepSeek's R1 significantly outperform other models on this benchmark.
The researchers selected the Sunday Puzzle as a benchmark due to its unique advantages. The quiz does not test for obscure knowledge; instead, it challenges models to solve problems that require insight and a process of elimination. Arjun Guha, a key figure in the research, emphasized this point:
“I think what makes these problems hard is that it’s really difficult to make meaningful progress on a problem until you solve it — that’s when everything clicks together all at once,” – Arjun Guha
Guha further elaborated on the nature of these challenges:
“That requires a combination of insight and a process of elimination.” – Arjun Guha
The quiz's design ensures that AI models cannot rely solely on "rote memory" to find solutions. This presents a unique challenge that demands genuine reasoning capability. The trade-off, however, is that reasoning models take a bit longer to arrive at solutions, with time differences ranging from seconds to minutes longer than simpler models.
The introduction of new questions every week ensures that AI models encounter truly unseen challenges. Guha remarked:
“New questions are released every week, and we can expect the latest questions to be truly unseen,” – Arjun Guha
This approach aligns with the research team's goal of developing benchmarks centered around general knowledge rather than specialized expertise. Guha stated:
“We wanted to develop a benchmark with problems that humans can understand with only general knowledge,” – Arjun Guha
Additionally, he noted the accessibility of reasoning skills:
“You don’t need a PhD to be good at reasoning, so it should be possible to design reasoning benchmarks that don’t require PhD-level knowledge,” – Arjun Guha
The study also explored whether AI models might "cheat" by being trained specifically on the publicly available quizzes. However, researchers have found no evidence of such practices occurring.
Leave a Reply