Researchers have ventured into an intriguing experiment by using riddles from NPR's Sunday Puzzle to create an AI benchmark. Premiered on NPR and hosted by Will Shortz, this long-running segment challenges listeners with problem-solving quizzes. The initiative aims to test the limits of AI's reasoning capabilities, as the puzzles are designed to thwart reliance on rote memory. The Sunday Puzzle is distinctly U.S. centric and presented in English, offering a unique set of challenges for AI models.
The newly developed benchmark encompasses approximately 600 riddles from the Sunday Puzzle episodes. The current standout model, o1, achieved a score of 59%, establishing it as the best-performing model on this benchmark. Meanwhile, the recently released o3-mini model, configured for high "reasoning effort," scored 47%. Such reasoning models, including DeepSeek's R1, significantly outperform others in this domain.
The challenges posed by the Sunday Puzzle provide a promising avenue to test AI's problem-solving abilities. Arjun Guha, one of the researchers involved in this project, emphasized the appeal of these puzzles due to their complexity.
“I think what makes these problems hard is that it’s really difficult to make meaningful progress on a problem until you solve it — that’s when everything clicks together all at once,” – Arjun Guha
Interestingly, these reasoning models exhibit a trade-off in terms of time taken to arrive at solutions. Unlike traditional models that rely on speed and memory, reasoning models may take additional seconds or even minutes to process and solve the puzzles. Guha elaborated on the objective of developing a benchmark accessible to all.
“We wanted to develop a benchmark with problems that humans can understand with only general knowledge,” – Arjun Guha
“You don’t need a PhD to be good at reasoning, so it should be possible to design reasoning benchmarks that don’t require PhD-level knowledge,” – Arjun Guha
Despite the advanced capabilities of these models, challenges remain. DeepSeek's R1 model sometimes provides incorrect solutions, openly admitting its frustration with certain problems.
“On hard problems, R1 literally says that it’s getting ‘frustrated,’” – Arjun Guha
The public availability of these quizzes raises concerns about potential "cheating," although there is no evidence supporting such behavior among AI models. Each week brings new questions, ensuring a fresh set of challenges for both humans and artificial intelligence alike.
“New questions are released every week, and we can expect the latest questions to be truly unseen,” – Arjun Guha
Leave a Reply