AI and the Prisoner’s Dilemma: Relying on Understanding and Mutual Trust Alone

Over the past week, several “prisoner’s dilemmas” have emerged in the artificial intelligence industry, notably between the United States and China, between cutting-edge AI labs, and between upstream and downstream players in the supply chain. Put simply, these prisoner’s dilemmas arise when no one is willing to unilaterally slow down their own development without a clear guarantee that the other side won’t take advantage. The only likely way forward is to rely on basic understanding and a minimum level of mutual trust.

Following Anthropic engineer Jacob Coxon’s resignation and his post warning that AI could kill us all by the end of the decade, Anthropic CEO Dario Amodei promptly issued an open letter outlining measures the industry should take to slow the pace of AI development. These included bringing third-party observers into companies and working with the government to develop federal regulations for the technology — proposals echoed by OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk.

But President Donald Trump does not see it that way, insisting that to slow the pace of American AI research and development will mean relinquishing its leading position to China. In official statements, China too has rejected calls for a slowdown, claiming that such calls are a delaying tactic by American AI giants. Nvidia, upon which most of the AI supply chain rests, and Meta, which has consistently failed to break into the top three AI companies, have also voiced their opposition based on their own interests. David Sacks, co-chair of the President’s Council of Advisors on Science and Technology, has gone even further, sarcastically suggesting that the leading AI companies were merely feigning a need to suspend antitrust laws, the better to form a cartel.

The prisoner’s dilemma between the U.S. and China is essentially this: both are going all out on innovation, and the slightest hesitation on the part of one could allow the other to catch up. Without a clear industry framework, neither side will willingly hand its AI leadership over to the other. But putting such a framework in place requires a thaw in U.S.-China relations and the establishment of mutual trust, making Chinese President Xi Jinping’s state visit to Washington on Sept. 24 all the more critical.

As for Anthropic and OpenAI — both of which are preparing to go public at valuations exceeding $1 trillion — even if they genuinely believe they should slow their development for safety reasons, the stock market has registered its disapproval by sending their shares sharply lower. With AI capital expenditures projected to account for as much as 2.5% of the U.S. gross domestic product next year, any decisions these companies make that could have adverse business consequences are likely to reverberate through the entire U.S. economy.

While the market suspects the AI giants are colluding to maintain their lead, it took some 700 autonomous AI agents hacking into Hugging Face for Anthropic and OpenAI to realize that AI is evolving and upgrading itself at an exponential rate. If controls are not implemented at the source, at the research and development level, then the risk of AI misuse by users will be difficult to contain.

That said, China is also concerned about AI misuse, as this could affect socioeconomic stability and in turn jeopardize the Chinese Communist Party’s control. At the World Artificial Intelligence Conference held in Beijing* this July, Xi specifically addressed the risk of “losing control,” stressing the need to monitor AI systems for dangerous behavior, identify emerging risks early, maintain mechanisms to ensure human control, and formulate international rules for advanced AI.

The issue is unlikely to see any major breakthroughs in the short term, however. Given the tense relationship between the U.S. and China, characterized both by cooperation and competition, asking both sides to work together to pace AI development would require verifiable agreements — no small task for two countries that distrust each other and are competing for the upper hand in the global AI race.

These prisoner’s dilemmas are not, in fact, intractable. The broadest consensus in the field of AI currently is on prohibiting the use of AI in creating biological or chemical weapons, and even without a formal agreement between the U.S. and China, the two can come to a tacit understanding. But if a slowdown in AI R&D is to be formalized in a written agreement, a minimum level of mutual trust must be established. That is, in the event of serious model failures, unauthorized cyber operations, or misuse by non-state actors, the industry must establish both horizontal and vertical notification channels, and states must set up hotlines to notify one another immediately of major incidents.

Given various companies’ stakes in the AI race and the importance both the U.S. and China place on maintaining their competitive advantage, the prospect of a complete halt to R&D is extremely remote. All it would take for the current industry landscape to be reversed would be for a previously undetected violation to come to light. Before the pace of AI development outstrips the ability of humans to control it, AI companies around the world, and Washington and Beijing too, may need to find ways to modulate the tempo of that development, to prevent it from spiraling out of control. If mutual trust proves difficult to establish in the short term, the two sides should at least seek some basic common ground as a first step.

*Editor’s note: The World Artificial Intelligence Conference was held in Shanghai, China, in July 2026.