Imagine spending months shaping a national policy to govern artificial intelligence, only to have that very technology quietly sabotage you from inside. That is exactly what happened in South Africa in April, when the country’s first attempt at a binding AI policy collapsed in less than three weeks. The Department of Communications and Digital Technologies published the Draft South Africa National Artificial Intelligence Policy on 10 April and opened it for public comment. Sixteen days later, the document was gone. It wasn’t killed by angry public submissions or a legal challenge. It was killed by its own footnotes. Journalists who started checking the references discovered something alarming: a number of citations simply did not exist. Some pointed to academic journals that had never been published at all. Others pointed to real journals, but the cited articles had never actually appeared in them. When the communications minister withdrew the draft, he was refreshingly honest. This wasn’t a technical glitch, he admitted. It was a failure of oversight. Generative AI had been used to help produce the policy, but nobody had properly verified the sources before the document went out. The credibility and integrity of the policy were compromised as a result. Much of the public commentary treated the episode as a joke. How embarrassing, after all, that a policy meant to regulate AI was itself undermined by AI. But a senior lecturer in cyber law who has written about these issues says that framing misses the point. Laughing off the incident as irony hides the deeper structural failures it exposed. The fabricated citations weren’t a silly accident. They were a warning.
To understand why, you have to understand what AI hallucination really is. Generative AI tools like ChatGPT and Grok are designed to produce text and images that sound or look convincing. But they don’t actually know what is true. A hallucination occurs when the system confidently produces something that is inaccurate, misleading, or wholly invented. It makes things up with the same tone of authority it uses when it is right. That makes hallucinations particularly dangerous because they are often difficult to spot unless someone takes the time to check every claim. We have already seen this happen in universities, where academics have been caught listing fake AI-generated sources in research. We have seen it in courtrooms, both in South Africa and elsewhere, where lawyers have submitted nonexistent cases as legal authority. And now we have seen it in the highest level of public policy. In the South African AI policy, the hallucination didn’t just invent a few random references. It manufactured a whole layer of seemingly credible African scholarly authority. Pretend experts were cited as real authorities. Respected authors, real academics with established reputations, were named in a false light. Real institutions, institutions that are recognised around the world as reliable publishers of academic research, were attributed evidence they never published. This is not just a citation problem. It is a breach of two essential kinds of integrity. The first is epistemic integrity, which means the research should have been conducted through reliable, ethical, repeatable methods that any reader could verify. That was absent. The second is information integrity, which means the public has a reasonable expectation that information from an authoritative government source can be trusted. That too was absent. And here is the most troubling part: the policy was supposed to govern these kinds of harms, yet it couldn’t even protect itself from them. The damage caused by generative AI is not limited to fake references. It includes fake images, fake videos, fake voices, and the weaponisation of real people’s likenesses through deepfakes. If a national AI policy can be quietly invaded by hallucinated authority figures, what does that say about the surveillance, identity theft, and disinformation that AI can unleash on ordinary citizens?
The South African draft policy, to its credit, was built on the idea of responsible AI governance. It drew on internationally accepted principles, including those from the Organisation for Economic Co-operation and Development and the Smart Africa AI Blueprint. These principles sound simple: accountability, transparency, and explainability. But they are not voluntary sweeteners. They are binding conditions. If a government policy demands them from AI developers and companies, then the same demands must apply to every institution that uses AI, including the government itself. That includes the production of public documents. The draft policy failed all three of its own core principles in the very process by which it was created. Accountability should have meant that the department could take full responsibility for what was in the document. Instead, there has been no clear explanation of how deep the fabrication goes. Which sections of the policy were affected? Which parts of the argument relied on false citations? Was the damage limited to the reference list, or did the hallucinated sources influence the substance of the policy’s main claims? The public does not know. The department has serious questions to answer on all of these points. Accountability does not mean firing an intern or saying “AI made a mistake.” It means standing in front of the country and laying out the full extent of the problem, honestly and without spin. It also means not rushing to revise the document until the department can demonstrate the same standards of evidence that the revised policy will ask of everyone else. This is a rare opportunity for the department to rebuild public trust and show what resilient, responsible governance actually looks like. But that will only happen if they embrace the mess rather than burying it.
Transparency and explainability demand even more. Transparency means disclosure. The public deserves to know which sections of the policy were materially affected by fake sources. The public deserves to know which AI tool was used, who used it, and at what stage of drafting it entered the process. Did the AI generate the literature review? Did it write the founding values? Did it synthesise public comments? Was it used to create the entire foundation of the policy, or only a minor part? None of this has been revealed. The department has not told the public which tool was used, who prompted it, or how the output was reviewed. Explainability takes this one step further. It requires that we can trace the reasoning behind the policy. It means being able to look at any section and understand why it says what it says, based on what sources and what logic. The hallucinated sources appear in the reference list, but without full disclosure, nobody can know which parts of the policy they were used to support. Nobody can know how deeply they shaped the foundational priorities and values of the proposed framework. This is especially frustrating because the public comment process offers a contrast. Public submissions have a verifiable record. You can trace where the information came from, who said it, and when. The same cannot be said for the policy itself. By its own standards, the department will have failed both transparency and explainability unless it conducts a section-by-section review and informs the public about which parts of the policy were affected, and to what extent. This is not bureaucratic overreach. It is the minimum required for any trustworthy government document in the age of generative AI.
So what needs to change? The retracted policy, for all its flaws, actually identified some important priorities. It recognised AI as a tool for inclusive economic growth, capacity development, and the protection of human rights. It also acknowledged that it was a “point of departure” and that different sectors would need different approaches. But the incident shows that the treatment of generative AI itself needs to change, not just in the production of policy documents but in the way the policy imagines the regulation of synthetic media. The draft treated deepfakes, misinformation, and AI-driven disinformation as future problems to be handled later, perhaps at a sectoral level. That is no longer acceptable. These are not niche technical issues. They are cross-cutting challenges to public trust. They require their own regulatory logic and governance mechanisms, built on cross-sectoral cooperation. A revised policy must treat synthetic media and information integrity as a structural pillar, not as a subcategory of innovation governance. The state is already living with these problems. It suffered one in the most direct way possible: AI generated fake evidence for the very policy that was supposed to manage AI. If the revised policy merely adds a note saying “we should be careful about deepfakes,” it will have learned nothing. Instead, it needs to create concrete rules about how generative AI is used in public institutions, when and how verification is required, and how the public will be protected when synthetic media is used to harm individuals, communities, or democratic institutions.
Finally, the revised policy must designate a specific mandate holder for synthetic media and information integrity. Right now, there is a patchwork of regulatory bodies with overlapping jurisdiction over digital content, identity harms, and information distribution. But what is missing is an agreed framework on definitions, remedies, and the practical steps to be taken when generative AI is used to spread misinformation and disinformation through fake sources, fake images, and fake voices. Who is responsible when a deepfake of a political leader appears hours before an election? What legal remedies does an ordinary citizen have when their face is grafted onto a pornographic video? How do we define synthetic media in a way that covers both obvious fakes and subtle manipulations? These questions remain unanswered. Creating a mandate holder does not require building new institutions from scratch. It requires political will. It requires the same ministries and agencies that already deal with digital content, communications, identity, and law enforcement to agree on a shared framework and coordinate their responses. It requires clear leadership. The article itself ends with a note that is worth remembering. The author, who drafted the piece, admits to using an AI tool called Claude to improve readability, but says they personally drafted, verified, and reviewed all of the substance and sources, and take full responsibility for the contents. That is a small, practical example of what responsible AI use looks like. It is not about avoiding AI. It is about using it transparently, verifying its outputs, and being willing to stand behind the final product. South Africa’s AI policy must demand nothing less from everyone else. And if it does, the embarrassing collapse of the first draft could become the foundation of something genuinely stronger.