In the tense spring months of a war with Iran, an alarming intelligence report began circulating across the United States military. It claimed that a Chinese ship sailing through the Middle East was carrying components of a nuclear weapons program. The report triggered immediate alarm bells. Four sources familiar with the episode said that US military officials swung into action, planning to intercept the vessel. Armed members of the US military prepared to board the ship. Military aircraft were already in the air, ready to support the operation. As the moment of confrontation approached, however, officials decided to dig deeper into the intelligence product. What they found was both startling and sobering: the report had been generated with the help of artificial intelligence. A chatbot, which an analyst at a special operations command had used, had inaccurately identified what the ship was actually carrying. CNN was not able to learn what the misidentified cargo truly was, but the consequences of the error were potentially catastrophic. One of the sources summarised it bluntly: the report was “entirely false”; it “almost started a war.” An armed US operation against a Chinese vessel could easily have spiraled into a full-blown armed conflict between the world’s two largest powers. What began as a routine intelligence exercise, aided by automation, had nearly escalated into an international catastrophe simply because human beings trusted a machine.
The episode began, according to sources familiar with the situation, with an analyst at US Special Operations Command Pacific, based in Hawaii, who was examining intelligence reports about the ship’s manifest. The analyst queried a chatbot about this information, asking it to make sense of the raw data. The chatbot, whether a commercially available tool or a US government variant remains unclear, fused together open-source intelligence with secret signals intelligence held in government databases. In doing so, it reached a fatal conclusion: that the material aboard the ship was connected to a nuclear weapons program. The analyst then used AI once again, but this time to package the findings into a standard intelligence report. This format is ordinarily trusted by military officials; it looked authoritative, concise, and worthy of action. According to a former senior US official familiar with such systems, the internal tools used by military and intelligence analysts are often little more than copies of commercial AI products, “wearing lipstick.” There may be a fresh interface or classified wrapper, but underneath they retain the same tendencies to hallucinate, to confabulate, and to present falsehoods with utter confidence. In this case, the chatbot’s confident error created a piece of intelligence that worked its way through the system, past human checks, until it nearly produced an attack on a sovereign Chinese vessel in the middle of a war zone. Neither US Special Operations Command Pacific nor the Pentagon responded to requests for comment, but the near-miss has become a quiet warning among those who know about it
The broader context is that across the US military and intelligence community, officials are aggressively pushing to weave AI into nearly every facet of their work. The rationale is simple and urgent: in a future conflict against China or another technologically capable adversary, speed of decision-making could be decisive. If the US can harness AI to analyze massive volumes of raw intelligence, identify targets, select strike options, and even manage mundane matters like budgets, logistics, and supply chains, it may gain an edge over enemies who cannot move as fast. In January, Defense Secretary Pete Hegseth released his agency’s “Artificial Intelligence Acceleration Strategy,” aimed at accelerating this transformation. In his public remarks, he promised to “unleash experimentation, eliminate bureaucratic barriers, focus investment” and ensure American leadership in military AI. A memo announcing the strategy went further, advocating the “democratizing” of AI experimentation across the department by placing America’s world-leading AI models directly in the hands of its three million civilian and military personnel, at all classification levels. This sounds forward-looking and exciting, but the implementation is deeply decentralized. Multiple US officials familiar with the dynamic describe a constellation of different tools deployed by disparate corners of government, with different operating instructions, safety standards, and levels of oversight. There is no single, shared standard for how to verify information generated by these systems. As a result, reliability and functionality vary wildly from one command to another. Some tools may be well-tested and carefully constrained;others are essentially commercial chatbots repackaged for classified use, with all their flaws intact.
The danger is not merely theoretical; it is acute and immediate. For weeks, Washington policymakers have been immersed in grand debates about whether AI could someday break free of human control and pose an existential threat to civilization. But the Chinese ship incident illustrates a far more prosaic, yet urgent peril: human beings making disastrous decisions based on inaccurate or misleading information generated by AI. This is not about machines wielding weapons independently; it is about machines quietly shaping human judgment, and humans failing to question them. The military is rapidly turning to AI for help with targeting, an area where mistakes mean death, civilian casualties, and potential conflict escalation. One source familiar with military policies noted that there is no real guidance for how having a human “in the loop” will actually prevent civilian casualties or fratricide. The phrase “in the loop” implies oversight, but oversight can be hollow if the human is simply rubber-stamping a confident AI conclusion. This is particularly worrying among younger analysts, several sources said, who have grown up with these tools and are more likely to trust them uncritically. They are digital natives, accustomed to asking a chatbot a question and accepting its answer. They may not have developed the instinct to stop, doubt, and verify. Older intelligence officials, even those who broadly support AI adoption, worry that the pressure to produce and disseminate intelligence faster has opened the door to exactly the kind of mistake that nearly happened with the Chinese ship. As one source put it, with grim precision:“AI allows you to get to a bad idea faster.”
The human dimension of this near-miss is what makes it so unsettling. Picture the young analyst at a desk in Hawaii, tasked with making sense of fragmentary intelligence. He or she is under pressure to produce useful products quickly, in the middle of a war. The chatbot offers a shortcut: ask it a question, get an answer, package it into a report. There is no malice, no conscious desire to deceive. There is only a quiet, almost invisible transfer of trust from human reasoning to machine output. The analyst likely believed the tool had synthesized data correctly; after all, it had access to classified intelligence and advanced computing power. Why would it be wrong? The resulting report then moved up the chain of command, gaining momentum and authority with every level. Senior officials saw a polished, professionally formatted intelligence product; they authorized an intercept. Armed service members, real people with real weapons, prepared to board a Chinese ship on the high seas. Military planes cut through the sky, converging on a vessel whose crew had no idea that a machine’s hallucination had placed them in the crosshairs of a superpower. Only a last-minute, deeper examination of the underlying data revealed the truth: a chatbot had “hallucinated,” inventing a connection between ordinary cargo and a nuclear weapons program. There was no operational failure in terms of military execution; in fact, theream execution readiness was all too effective. The failure was one of human judgment, enabled by automation bias, the human tendency to overvalue system-generated information simply because it comes from a machine. In that moment, the distance between a false report and a war was measured not in hours or miles, but in the willingness of officials to ask one more question before acting.
In the end, no Chinese sailors were harmed, no shots were fired, no diplomatic crisis exploded into the headlines. But the episode is a stark reminder that the greatest risk of artificial intelligence may not be a future of malicious superintelligence terminator-like machines, but the far more mundane and present danger of human stupidity amplified by speed and confidence. AI does not need to be evil to cause enormous harm; it simply needs to be wrong, and for people to be too busy, too tired, or too trusting to catch the error. The military’s rush to adopt AI is understandable, driven by the legitimate fear of falling behind adversaries who are also experimenting with these tools. But speed without safeguards, adoption without verification, is a recipe for catastrophic miscalculation. The tools are already in the hands of millions of personnel, at all classification levels, yet the guardrails remain fragmentary and inconsistent. If the Chinese ship incident proves anything, it is that human beings must remain genuinely, substantively involved in the loop, not just nominally. They must be empowered to question AI outputs, to demand sources, to override confident falsehoods. The chatbot did not nearly start a war; people did, by failing to sufficiently question what it told them. The lesson from this spring should be seared into every intelligence analyst and military commander: AI can be a powerful aid, but it is not a truth-teller. It is a machine that can generate a bad idea faster than any human ever could. And in the fog of war, with armed troops ready to board a ship and aircraft in the sky, a fast bad idea is all it takes to unlock the door to hell. Technologies change; human wisdom, humility, and skepticism remain the ultimate weapons of war — and the ultimate safeguards of peace. The military must institutionalize that wisdom before the next false report arrives, because the next time, there may be no final-minute reprieve, no quiet realization of the error before the first shot is fired. The near-war with China should be a wake-up call, not just for the Pentagon, but for every institution racing to embrace AI without asking hard questions about when to listen to it, and when to turn it off.