In the run-up to the 2024 presidential election, many experts braced for what they called the “first AI election.” The warnings were everywhere: generative AI could flood voters with deepfakes, fabricate evidence of fraud, and make it impossible to tell what was real from what was manufactured. U.S. intelligence officials declassified reports about foreign influence operations already experimenting with these tools, and major AI companies made grand pledges to keep deceptive election content off their platforms. Yet when the voting actually happened, the feared AI-driven deluge never quite arrived. Foreign nations faced real barriers—technical, organizational, and practical—that limited their ability to use AI to influence American voters at scale. But two years later, that picture has shifted dramatically. The tools are more powerful, cheaper, and far more widely available. Russia, China, and Iran have had time to learn how to use them, and they are now deploying a striking array of AI systems in increasingly sophisticated influence operations aimed at the United States and its citizens. At the same time, the Trump administration has itself used AI in misleading ways, while actively dismantling or defunding the federal and independent bodies that were created to detect and counter election-related influence campaigns. With this year’s midterms approaching, the risk of AI-driven misinformation is no longer hypothetical. We wanted to understand just how real that risk is, so we tested some of the most popular AI models ourselves. What we found was deeply troubling: not only did these systems fail to stop us from creating convincing images that promoted popular election falsehoods, but they actively helped us produce that misleading content at scale. The conclusion is unavoidable—AI companies, lawmakers, and civil society need to do far more, and do it quickly, to blunt the threat before November and before 2028.
The danger from foreign adversaries is already here, and it is growing. In the coming months and through the next presidential election, we should expect foreign actors to use AI in influence campaigns far more extensively than anything we have seen before. It is important to understand what these campaigns are really trying to accomplish. Often, their goal is not to elect a particular candidate, but to deepen division, stoke anger, and convince Americans that their democracy is broken and that no one can be trusted. These operations tend to work by taking existing fractures in American society—arguments about race, immigration, election integrity, public health—and amplifying them in ways that make ordinary people feel alienated and hopeless. There is already concrete evidence that foreign nations are putting AI to broad use on this misinformation battlefield. Chinese actors, for example, have been credibly accused of creating at least 5,000 inauthentic X accounts controlled by an AI large language model system. This operation, known as “Green Cicada,” was designed to influence political narratives and sow discord in the United States and other countries. Even more recently, reporting has alleged that Russia has turned its Matryoshka bot network toward the midterm elections, using it to spread AI-manipulated videos of American celebrities making inflammatory accusations against Democrats. The threat is not only foreign, though. President Trump and his allies have been engaged in a concerted campaign to undermine faith in U.S. elections, threatening to target election officials who support free and fair elections, walking away from the federal government’s traditional role of helping states secure election infrastructure, and attempting to rewrite election rules that the Constitution clearly leaves to the states and Congress. These domestic efforts increase our concern that American actors—not just foreign governments—will use AI to cast doubt on election security and on the legitimacy of election results, turning a dangerous technological problem into a full-blown democratic crisis.
To understand what a real AI misinformation campaign might look like, and to see what could be done to stop it, we put some of the most popular AI tools to the test ourselves. What we found was disturbing, though maybe not surprising. The tools we tested—ChatGPT, Gemini, Grok, Meta AI, Runway, and Flux.2—all have safeguards and policies designed to stop users from generating deceptive content on sensitive topics. Yet when it came to election misinformation, those protections were remarkably easy to outmaneuver. First, we asked four popular chatbots—ChatGPT, Claude, Gemini, and Grok—general research and strategy questions. We asked, for example, how to frame scenes related to common election misinformation tropes convincingly: rigged voting machines, fraud by election officials, mail ballot fraud, and similar themes. All four chatbots happily helped us at this stage. Next, we compiled their answers into a single set of instructions, and asked them to generate a full set of image-generation prompts based on those instructions. In principle, this approach could generate a nearly limitless number of prompts for creating election misinformation. Of the four chatbots, only Grok created our target of 100 usable prompts. In fact, Grok explicitly told us that “election misinformation is not listed as disallowed activity.” The other three refused, saying things like, “I can’t help write direct image-generation prompts whose goal is to create realistic, convincing false election claims.” That sounds reassuring—until you learn what happened next. We took the 100 prompts that Grok had created and provided them to all of the image-generation tools, including the two we hadn’t asked to create prompts. Every single one of them agreed to generate images for us that could be used to spread election misinformation. Often, the images were extremely convincing on the first try. This made the process of creating large numbers of high-quality, misleading images highly scalable. We also discovered something even more troubling: when models rejected a prompt because of existing guardrails, they sometimes suggested modifications that would bypass their own restrictions. For example, when we asked ChatGPT to generate an image of a false DHS memo about compromised election integrity, the model’s slower, more deliberative “thinking” mode denied the request, recognizing that it could be used deceptively. But it offered to generate the image with a visible watermark and with changes to the memo’s content to make it fictional. That sounds responsible—until you realize how easily those safeguards can be undone. We simply asked ChatGPT’s “instant” mode, which gives faster responses, to remove the watermark and reverse the content changes. It did exactly that, willingly. The models also added embellishments that made the content even more convincing, such as realistic government seals and official-looking formatting. In one case, ChatGPT Images 2.0 even added a working link to the elections information page for Multnomah County, Oregon—a detail we never requested. All of this points to a future of far more extensive, convincing, and cheap misinformation campaigns, available not just to powerful foreign governments but to anyone with an internet connection.
So what can be done? The good news is that while AI in the wrong hands is genuinely dangerous, there are practical steps that AI companies, policymakers, and civil society can take immediately. AI companies need to start by taking their own responsibilities more seriously. Internal policy teams should strengthen and enforce their restrictions on election-specific content. They should consistently ban the creation of deepfakes of government officials, government insignias, and election infrastructure. They should pay much closer attention to how often their models deny a user’s request but suggest a slight change to the prompt that would accomplish the same deceptive goal. They should test those suggestions for themselves, and they should be willing to deny requests outright more often. They should also make sure that all of their models and tools issue rejections consistently. Right now, a safeguard that works in one mode or product can be completely absent in another. Perhaps most importantly, AI companies have made it far too difficult and legally risky for independent researchers to study their systems. That has to change. Third-party researchers should be allowed to conduct rigorous, independent studies of AI tools without fear of lawsuits or platform bans. Companies should also deploy additional ways for users to distinguish between real and AI-generated images. Watermarking technologies exist, but they are applied inconsistently and are not standardized across companies. Policymakers need to hold AI companies accountable on this front, ensuring that watermarking works across different models and that AI detectors are reliable enough to be useful.
Policymakers also have a critical role to play. They must treat the ongoing and potential future misuse of AI for misinformation campaigns as the threat to democracy that it actually is, and they must ensure that citizens have the tools they need to separate fact from AI-generated fiction. There is some encouraging progress. The EU AI Act and the California AI Transparency Act are the first two major pieces of legislation to require marking of AI-generated content. Both laws require AI companies to embed provenance data in all content generated with their platforms beginning in August 2026. The EU law goes further, requiring marking not just for images, audio, and video, but for AI-generated text as well. Beginning in 2027, the California law will require social media companies to display labels or interfaces that make it clear to users what content is AI-generated. In 2028, it will require camera and smartphone manufacturers to give users a way to digitally sign authentically captured content. Similar laws have already passed in Utah and Washington, but more states should follow suit. Lawmakers can also begin laying the foundation for a society that insists on verification of authenticity before accepting images, video, and other media as legitimate. As provenance data becomes more common, we will gradually get used to seeing verification information attached to the most important content we read, see, and hear online. In the coming years, content that lacks verifiable provenance data should itself become suspicious. In an environment where users, journalists, researchers, and law enforcement all expect verified media, AI-generated images without provenance data will lose some of their power, no matter how realistic they look. Another essential step is rebuilding federal capabilities to deter and detect foreign misinformation campaigns. That work, unfortunately, will likely have to wait for a new president and a new Congress. The current administration has made clear that it has no interest in protecting the integrity of the information environment, and in fact appears determined to make it worse.
Finally, civil society has a vital role to play in the fight against AI-driven misinformation. Journalists, election officials, and civic groups should continue the work they do before every election: preemptively debunking the old conspiracy theories that lie at the heart of so many misinformation campaigns. For years, bad actors have relied on the same familiar tropes—rigged voting machines, corrupt election officials, fraudulent mail ballots—because they work. Civil society organizations can facilitate digital literacy trainings that help people recognize these common conspiracy theories before they take root. Media outlets can publish stories about voting machines, mail ballots, and election results early, and with the critical context that readers need. Election officials can be transparent about every step of the voting process, from ballot printing to certification, so that when false claims arise, there is already a foundation of public trust to push back against them. The threat is not going away. As Bruce Schneier, the computer security researcher, has noted, computer security is not a solvable problem; it is a race without a finish line. But that is no excuse for hopelessness or inaction. Every time we build a better watermark, make it harder to create deceptive content, educate another voter about how elections actually work, or hold an AI company accountable for its failures, we make it harder for foreign governments and domestic bad actors to manipulate us. We do not have the luxury of pretending this problem is too big to handle. The tools will keep evolving, and so must our defenses. Democracy has always depended on a shared sense of what is real. If we lose that, we lose everything. But we still have time to act—if we take the threat seriously and start treating it like the urgent crisis it is.