Josh You, an analyst at Epoch, recently published a counter-intuitive and thought-provoking analysis. He’s particularly interested in the performance gains realized through standard model training and reinforcement learning in artificial intelligence models. Residing in Manhattan with his partner, who works as a music therapist, You’s insights shed light on the current landscape of AI development and OpenAI’s trajectory.
Based on You’s analysis, the performance breakthroughs from typical AI model training have been remarkable, improving fourfold per year. At the same time, reinforcement learning has experienced even more explosive growth, with performance doubling every three to five months. All of these improvements are a sign of a healthy research field, but they leave us with some disturbing questions about sustainability and future advances.
OpenAI—one of the largest and leading organizations in AI development—has released reasoning-centered models like o3 and o4 mini. These models have achieved state of the art gains on many AI benchmarks, most notably in their math and coding abilities. OpenAI has its eyes set on the future, with some wildly ambitious goals. They focus on a transition to reinforcement learning, convinced that will produce even bigger breakthroughs.
To further this move, OpenAI has pledged to spend much more computing power on reinforcement learning. For us to have you improve your capabilities, we provided roughly ten times as much computing power as we used to train the o1 model when training the o3 model. Epoch’s analysis indicates that a large portion of this computational spend ended up being spent on reinforcement learning projects.
These powerful AI reasoning models have seen tremendous success to date. Josh You’s recent analysis sounded the alarm that this momentum could begin to flag. He expects the pace of advances from these reasoning models to taper off in under a year. By 2026, he predicts it will catch up with the overall frontier.
“If there’s a persistent overhead cost required for research, reasoning models might not scale as far as expected,” – Josh You
This caveat concerning the scalability of reasoning models raises important questions about the scalability of reasoning models in the wake of these possible limitations. Other researchers, including Dan Roberts at OpenAI, are releasing initiatives that emphasize reinforcement learning. This new focus has led to much spirited debate from all sides in the AI community about what’s the right mix of training methodologies.
Leave a Reply