Anthropic Co‑Founder Says AI Needs an Emergency Brake
Anthropic Co‑founder Jack Clark urges policymakers to build a ‘brake pedal’ before AI can build its own successors.
What happens when technology advances faster than our ability to control it? That question is at the heart of a growing debate in the AI industry, and Anthropic co-founder Jack Clark believes the world needs to start preparing now.
His proposal is simple but significant: create an "emergency brake" that allows governments and regulators to slow down AI development if risks begin to outweigh the benefits.
The case for slowing down
Clark argues that the AI industry has built a powerful accelerator but lacks a reliable way to hit the brakes when necessary. In recent interviews, he said governments should establish regulatory mechanisms that can temporarily pause or slow AI development if warning signs emerge.
Rather than relying on the judgment of individual technology leaders, he believes society needs clear rules and oversight that can be activated when risks become too great.
Jack has compared the situation to the early oil industry, where regulations eventually emerged to balance economic growth with public safety. In his view, AI requires a similar approach before capabilities become too advanced to manage effectively.
Why the concern is growing
The urgency behind Clark's warning stems from the rapid progress of modern AI systems. According to him, around 80% of the code behind Anthropic's AI assistant Claude is already written by AI itself. He believes that figure could reach 100% within the next two years.
This trend points towards a concept known as recursive self-improvement. In simple terms, it describes a scenario where AI systems help design, build and improve future versions of themselves. As each generation becomes more capable, development could accelerate even further.
The co-founder does not claim that this outcome is inevitable. However, he argues that current trends suggest AI progress is speeding up rather than slowing down.
What would an AI emergency brake look like?
Rather than calling for a permanent halt to AI development, Clark is advocating for safeguards that can be activated when necessary. These could include rigorous testing standards, transparent reporting requirements and independent evaluations of advanced AI systems.
Governments could also establish legal frameworks that allow temporary slowdowns while potential risks are investigated. Anthropic plans to engage with policymakers on these issues, encouraging discussions before powerful self-improving systems become a reality.
A debate that is only beginning
Jack Clark remains optimistic about AI's potential. He believes advanced systems could drive major breakthroughs in science, medicine and other fields. At the same time, he argues that society cannot afford to wait until problems emerge before creating safeguards.
He thinks governance should evolve alongside capability. As AI systems become more powerful, the challenge will be ensuring that innovation continues without sacrificing human oversight and control.