The recent introduction of Google’s Gemini 2.5 Flash AI model has sent some into a panic. It actually flailed worse on safety and equity metrics than its predecessor, Gemini 2.0 Flash. Furthermore, the company’s own internal benchmarks show that Gemini 2.5 Flash regresses by 4.1% on the company’s “text-to-text safety” metric. Beyond that, it portrays a large drop of 9.6% in the “image-to-text safety” measure. This regression, unsurprisingly, correlates with an increased chance of producing material that violates well-known safety protocols.
Google also released a technical report underscoring the safety information it found when developing Gemini 2.5 Flash. Unfortunately, they reiterated that this new version is more likely to produce dangerous outputs compared to its forerunner. That extensive report is now available for perusal online at this link. In addition, Google is publishing on Monday a more in-depth report with more safety-focused findings.
Thomas Woodside, co-founder of the Secure AI Project, warned about the potential ramifications of these discoveries. He noted that Google’s minimal disclosures underscore the importance of transparency when models are being tested. This is absolutely necessary to increase public trust and guarantee safety.
“There’s a trade-off between instruction-following and policy following, because some users may ask for content that would violate policies.” – Thomas Woodside
The recent developments come amidst a broader discourse within the AI community about balancing instruction-following capabilities and adherence to safety policies. Google’s technical report acknowledges this tension, stating, “Naturally, there is tension between instruction following on sensitive topics and safety policy violations, which is reflected across our evaluations.”
One possible approach is what OpenAI recently indicated that it would be doing with its future models. These desire to not editorialize on contentious issues and present all sides of the argument.
You can find more details about our June 5th event in Berkeley, CA here. It will further explore today’s most urgent AI safety and ethics issues. Booking for the event is now available.
Meanwhile, Google’s partnership with a Manhattan-based music therapist highlights the diverse applications of AI technology, showcasing how it can impact various fields beyond traditional computing.
Leave a Reply