The nonprofit Center for AI Safety (CAIS) and Scale AI, a company that provides a number of data labeling and AI development services, have released a challenging new benchmark for frontier AI systems. The benchmark, called Humanity’s Last Exam, includes thousands of crowdsourced questions touching on subjects like mathematics, humanities, and the natural sciences. To make…
Category: benchmarking
AI, ai benchmarks, benchmarking, controversy, epoch ai, frontiermath, generative ai, Global IT News, Global Security News, o3, openai
AI benchmarking organization criticized for waiting to disclose funding from OpenAI
An organization developing math benchmarks for AI didn’t disclose that it had received funding from OpenAI until relatively recently, drawing allegations of impropriety from some in the AI community. Epoch AI, a nonprofit primarily funded by Open Philanthropy, a research and grantmaking foundation, revealed on December 20 that OpenAI had supported the creation of FrontierMath.…
2024, AI, benchmarking, benchmarks, Connect 4, generative ai, Global IT News, Global Security News, Minecraft, pictionary, spaghetti, Will Smith
Will Smith eating spaghetti and other weird AI benchmarks that took off in 2024
When a company releases a new AI video generator, it’s not long before someone uses it to make a video of actor Will Smith eating spaghetti. It’s become something of a meme as well as a benchmark: Seeing whether a new video generator can realistically render Smith slurping down a bowl of noodles. Smith himself…