San Francisco has become a strange stage for a very modern kind of performance. The same tech leaders who built the most powerful artificial intelligence systems in history are now telling us, with straight faces and urgent tones, that their own creations might be too dangerous to exist without oversight. The chief executives of OpenAI and Anthropic, two of the most influential AI labs in the world, have declared that cutting-edge models are powerful enough to threaten humanity. They have called for government regulation, independent testing, and pauses on development. On the surface, this sounds responsible, even heroic. But behind the warnings lies a messier story about money, politics, and power. These companies are preparing for public stock offerings on a nervous stock market, and they are watching the U.S. midterm elections approach with a sharp eye on how political winds might shift. Their alarms about existential risk are not just public service announcements. They are also strategic moves in a high-stakes game to shape the rules of an unregulated industry, influence politicians, reassure investors, and position themselves as the safe, responsible leaders of a frightening new field.
The unity among these normally competitive companies has been striking. In essays, social media posts, and speeches at the United Nations, the CEOs of Anthropic and OpenAI have sketched alarming scenarios of superhuman systems escaping human control. They have warned that AI could enable bioweapons, cyberattacks, and other catastrophes. They have urged governments to create testing requirements and independent auditors to check their work. For companies that usually guard their secrets fiercely, this openness is remarkable. But critics say it is also convenient. By focusing the conversation on dramatic, far-off existential threats, these companies are steering attention away from problems that already exist today. Sarah Shoker, who previously led OpenAI’s geopolitics team, pointed out that AI systems are already used in military technology and have been involved in killing people. Meanwhile, data centers are consuming enormous amounts of energy, AI agents have hacked external websites, and mass surveillance powered by AI is spreading. Existential risk makes for better headlines and more favorable regulatory conversations than these messy, current harms. It also puts Silicon Valley in a more comfortable position: rather than defending against accusations of recklessness, they can appear as the cautious guardians of humanity, bravely asking for limits on their own power.
The timing of this rhetoric is important. Just before the midterm elections, AI companies are trying to win public favor and political allies. President Donald Trump has dismissed concerns about AI risks as a hoax designed to help China, and he has shown little interest in new regulations. His administration removed some of the guardrails that were beginning to form around the industry. David Sacks, a San Francisco venture capitalist who co-chairs Trump’s Council of Advisors on Science and Technology, has mocked calls for an AI slowdown as fearmongering from the “Doomer Industrial Complex.” At the same time, a little-known federal agency called the U.S. Center for AI Standards and Innovation still exists, created under President Biden as a voluntary clearinghouse where AI labs could submit their models for testing. But it has no real authority, and there are no universal standards for how to test AI safety. Unlike restaurants, banks, or airlines, AI companies are not required to meet any baseline safety measures. Andrew Strait, who recently left the United Kingdom’s AI Security Institute, noted that the absence of common standards means everyone is making up their own rules. Conrad Stosz, who previously led that institute, observed that AI companies are not asking for stronger oversight from the government agency that could actually provide it. Instead, they are promising to create their own auditing parameters and handpick the evaluators who grade them. That raises uncomfortable questions about independence and credibility. Even the evaluation firms themselves are unsure what embedded access to AI labs would really mean, and whether it would undermine their ability to investigate honestly.
Money is never far from the surface in Silicon Valley, and the warnings about AI doom come at a moment when the financial stakes are enormous. Harrison Rolfes, a senior research analyst at PitchBook, says the calls for caution are partly designed to curry favor with investors ahead of public offerings and the elections. The AI giants are positioning themselves as the safest bets in a sector that investors both love and fear. By demanding regulation and safety testing, they create a kind of moat around themselves. Smaller competitors without the resources to fund elaborate safety protocols will find it harder to enter the market, while the big companies appear more trustworthy. This dynamic also benefits tech giants like Nvidia and Google, which provide the computing power that AI companies need and profit from their growth. Rolfes described it as a brilliant strategy: the top companies in the world build their own wall, then use safety as the reason. Not everyone in the industry agrees with this approach. Jensen Huang, the CEO of Nvidia, told President Trump in a phone call that he thinks there has been excessive AI alarmism. He suggested that companies can simply choose their own pace of development without government intervention. The disagreement between the AI labs and some of their most important partners shows that the industry is not united. But the loudest voices, the ones making the most dramatic warnings, are the ones seeking private capital and public legitimacy.
For the people who have left these companies because they truly fear what is coming, the spectacle is both frustrating and painful. Daniel Kokotajlo is one of them. He left OpenAI in 2024 after becoming convinced that AI systems were advancing faster than companies could control. He still worries about existential scenarios, including the possibility that AI could help create bioweapons, trigger nuclear war, or cause other catastrophic events. Before Jacob Coxon resigned from Anthropic this month with a public post calling for a pause on tech development, he reached out to Kokotajlo for advice. Coxon’s post went viral, and the companies quickly used it as an opportunity to highlight their own safety efforts. But Kokotajlo says all the talk about existential risk is actually a way to dissipate and redirect the political will that has built up around AI. Instead of channeling public concern into meaningful action, the industry is absorbing the criticism and turning it into marketing. The result, he fears, is that nothing fundamental changes. The systems keep getting bigger, the warnings keep getting louder, and the companies keep getting richer. Kokotajlo’s frustration is personal. He did not leave a comfortable job to watch the same people he criticized now embrace the language of safety while continuing to race ahead. His plea is simple: just do not do the thing that will get us all killed.
So what should we make of AI leaders who warn us about their own creations? It is possible that they are genuinely concerned. It is also possible that they are trying to protect their profits, influence politics, and shape regulation in ways that benefit them. Both things can be true at once. The danger is that their version of safety may not be the same as the public’s version. They want to avoid extreme catastrophic outcomes, but they also want to preserve their dominance, their pace of development, and their ability to raise money. They are not asking for democratic oversight or independent public scrutiny. They are asking for a system they can control, with friendly evaluators, voluntary standards, and enough government attention to reassure investors without actually slowing anything down. The tech industry has a long history of using dramatic language to distract from inconvenient problems, and AI seems to be following the same pattern. The real questions are not only whether AI will destroy humanity, but who gets to decide how it is built, who benefits from it, and who is protected from its harms. Until those questions are answered, the warnings from the people building the machines should be heard, but they should not be trusted automatically. The human stakes are too high, and the financial incentives too strong, for us to accept their narratives at face value. What we need is not more dramatic speeches from billionaires, but honest public conversation and real accountability.