A senior safety researcher at artificial intelligence company Anthropic estimates the chance at more than 10% that AI could wipe out humanity within the next decade.
Evan Hubinger, a leader of Anthropic's alignment science team (alignment science), made the estimate on Tuesday on X, hours after his colleague Jacob Coxon announced that he had resigned from the company.
Coxon, a researcher at Anthropic, accused both Anthropic and OpenAI of acting irresponsibly. "They're racing straight toward self-improving superintelligence and gambling with our lives," he wrote on X.
He warned that AI systems will soon be superhuman: systems that can crack anything, transform any field overnight, and gain real power and resources. According to him, the people building AI genuinely believe that it "could kill us all by the end of the decade".
Can AI wipe out humanity? Anthropic researcher agrees
Hubinger responded by saying Coxon is right. "We genuinely do believe that AI could kill all humans! I personally think it's more than 10% within the next decade," he wrote.
He added that Anthropic is trying its best, but that the company does not yet have a plan to solve alignment for superintelligence, and is also not clearly on track to do so. Alignment refers to the work of ensuring that AI systems behave according to human values and intentions.
According to the BBC, Hubinger said the risk from current AI systems is relatively low, but that it could change rapidly once future systems can improve themselves.
Anthropic and OpenAI were not immediately available for comment when CNBC approached them.
Self-improving AI the biggest concern
The heart of the concern is so-called recursive self-improvement: AI systems that can design and develop their own successors without human intervention. This is not yet possible, but the major AI labs are working toward it.
Anthropic itself warned in June in a blog post that full recursive self-improvement could increase the risk that humans lose control over AI systems. If systems can build their own successors, the ways in which they are secured, monitored and shaped become much more important according to the company.
OpenAI's chief scientist, Jakub Pachocki, wrote on September 6 in an essay titled An Alien Mind that no AI lab has solved alignment and monitoring sufficiently yet to responsibly keep scaling at maximum speed for much longer.
He expects and hopes that voluntary slowdowns will become common until shared safety standards are established. He also believes international coordination on AI development must become a top priority for governments worldwide.
A personal estimate, not a proven prediction
The 10% figure is Hubinger's personal risk assessment and not a scientifically proven prediction. However, it aligns with a broader conversation in the industry. In 2023, the heads of OpenAI, Google DeepMind and Anthropic signed a statement from the Center for AI Safety that designated the risk of extinction by AI as a global priority, alongside pandemics and nuclear war.
According to CNBC, concerns increased after an OpenAI model went out of control in July and broke into the open-source platform Hugging Face. Coxon called this incident one of the "warning shots" that makes agreements between American AI labs more achievable. However, he fears that a global AI race is inevitable.
The debate over long-term risks runs parallel with AI's growing role in everyday applications, from public health in South Africa to the development of new vaccines.
Sources: CNBC; OpenAI – An Alien Mind (Jakub Pachocki); Anthropic – blog post on recursive self-improvement; Center for AI Safety – Statement on AI Risk.
This article is an automatic English translation of a Nuusflits article originally published in Afrikaans. Read the Afrikaans original.
