Leaders at several major artificial intelligence companies are pushing for stronger safety rules and coordination, aiming to slow the pace of frontier model development and manage potential risks. Anthropic CEO Dario Amodei recently advocated for slowing AI progress, embedding safety evaluators in research labs, and working with government to set standards. Sam Altman and Elon Musk also supported these proposals, signaling growing concern among top industry leaders. Amodei has suggested targeted antitrust exemptions to allow competitors to coordinate on safety work, a proposal that has also raised concerns about competition. On September 15, Federal Trade Commission Chairman Andrew Ferguson said he was skeptical of calls for antitrust exemptions from AI companies seeking new regulation.
Warnings from within the industry are becoming more urgent and public, with some researchers expressing fear that AI could pose existential threats to humanity, according to Reuters and Fast Company. Former Anthropic researcher Jacob Coxon said that people working on AI privately believe it could end human civilization before the decade is over. Evan Hubinger, Anthropic’s alignment science lead, publicly backed Coxon’s concerns and estimated a greater than ten percent chance of such an outcome within ten years. Hubinger said researchers do not yet have a plan to solve the alignment problem for a future superintelligent system and are not clearly on track to develop one.
Amodei also warned of AI agents conducting unauthorized cyberattacks and the risk of powerful systems escaping human control. Some critics have questioned whether these calls for regulation are more about protecting market positions than safeguarding public safety. Economist George Stigler explained that businesses often seek regulation to limit competition and protect their own interests. This pattern of industry leaders pushing for rules that restrict entry is not new, according to Stigler’s analysis. The current debate includes proposals for industry coordination and targeted antitrust exemptions, which have prompted concerns about reduced competition.
Amodei’s plan includes independent evaluators with employee-like access to advanced AI labs and coordinated safety standards for frontier models. Amodei has said he is not calling for a halt to model training or technical progress. Instead, he wants companies to take more time to test and safeguard increasingly capable systems. Critics argue that complex compliance rules could make it harder for smaller companies to compete, while supporters say common standards are needed to reduce catastrophic risks. Open-weight AI models allow users to download and modify model weights rather than relying only on a company-controlled service. A regulatory system that requires centralized oversight would be difficult to apply to widely distributed open models.
The idea of pausing or slowing some AI development has gained traction among parts of the industry and some lawmakers, although major technology leaders remain divided. Nvidia CEO Jensen Huang and Meta CEO Mark Zuckerberg have opposed an industry-wide slowdown while supporting safety work by individual companies. In recent weeks, warnings about AI risks have shifted from outside experts to employees within leading companies, according to a report from fastcompany.com. Former Anthropic researcher Jacob Coxon recently resigned and shared his concerns online, drawing over 171 million views. Evan Hubinger, Ethan Perez, and Samuel Marks from Anthropic also publicly expressed similar concerns.
AI models are now being used for coding, research and other work that can improve the software and infrastructure used to build future AI systems. These tools are helping companies like OpenAI accelerate their own research and development efforts. However, this same technology is also being used to speed up the development of even more advanced AI systems. Reuters reported that concerns about recursive self-improvement are a major reason some industry leaders now want more time for safety work.
Meanwhile, global leaders are debating how to approach AI safety and cooperation between nations, Fortune reported. After a viral post from an Anthropic researcher, there has been discussion about coordinating with China to slow AI development. Reuters reported earlier this month that U.S. and Chinese officials were preparing for dedicated talks on AI safety and AI-directed cyberattacks. President Donald Trump and Chinese President Xi Jinping are set to meet in September, but both have rejected the idea of slowing AI progress. Trump criticized Amodei’s proposal and emphasized the need to maintain U.S. technological leadership. The United States is also limiting exports of advanced AI chips to China, as part of broader efforts to contain Chinese AI advancements.
George Chen of The Asia Group told Fortune that Xi sees AI as a major opportunity to reshape the global balance of technological power. Talks about AI safety and cooperation between the U.S. and China are ongoing but not expected to lead to immediate agreements. Both countries are also discussing principles to prevent misuse of AI by non-state actors like financial system attacks. The Trump administration also created a voluntary framework in June for pre-release cybersecurity reviews of frontier AI models.
The idea of an AI slowdown has not gained traction in China, where many believe the government should manage safety risks. In contrast, the U.S. tech industry generally supports closed models developed by companies like Anthropic and OpenAI. China, on the other hand, has focused on open-weight models that are more accessible and affordable. Shengyu Liu, an engineer at DeepSeek, believes frontier intelligence should be available to everyone in an open and inexpensive form.
The industry’s focus on AI safety and saving humanity has not resonated in the same way with much of the Chinese public, although Chinese officials and AI safety researchers have raised their own concerns about risks such as cybersecurity and loss of control. Overall, concerns about AI risks are growing, but perspectives differ between U.S. companies and Chinese authorities.
IMAGE: 24/05/2023. London, United Kingdom. The Prime Minister Rishi Sunak meets with Demis Hassabis, CEO DeepMind, Dario Amodei, CEO Anthropic, and Sam Altman, CEO OpenAI, in 10 Downing Street. Picture by Simon Walker / No 10 Downing Street. Photo: UK Prime Minister / Wikimedia, taken 2023-05-24, CC BY 2.0