Artificial intelligence has become a quiet but powerful presence in modern life. It recommends songs, writes emails, helps doctors interpret scans, and now it is showing up in courtrooms across the country. For lawyers, the promise of AI is enormous: legal research that once took hours can be done in seconds, initial drafts can be generated instantly, and the cost of preparing a case can fall dramatically. But a case involving the San Antonio Independent School District is a stark reminder that AI does not actually know anything; it only knows how to sound convincing. The case began when a student brought a small knife to school. Administrators sent the student to a disciplinary alternative education program, and the student’s mother filed a lawsuit against the district, arguing that the punishment was too severe for the situation. The case did not go her way at first, so her attorney appealed. The appeal relied on eight case citations. Four of those citations were apparently invented by an artificial intelligence program. In an opinion that caught the attention of the legal world, Justice Lori Valenzuela of the 4th Court of Appeals wrote that the four cases “appeared to be hallucinations created by AI.” The attorney, who practiced in Miami, was disciplined, and the law firm was warned not to repeat the behavior. The appeal itself was not dismissed, but the warning echoed far beyond one office. To understand what happened, it helps to know what “hallucination” means in the world of artificial intelligence. It is not a dream or a vision. It is a failure mode in which the software produces something that looks real, sounds real, and is entirely false. In a legal context, that means made-up court rulings. The AI did not misplace a comma or misunderstand a quote; it created case law from nothing, complete with the formal language of judicial opinions. That is why the San Antonio case is so unsettling. It reveals how easily AI can slip its way into the very institutions where truth is supposed to matter most.
To understand why this case matters, one has to look closely at what the appeals court actually found. The attorney’s brief listed eight cases in support of the mother’s argument. At least some of those were legitimate, but four had a strange quality. They were formatted like real opinions, with case names that sounded plausible, citations that looked official, and legal reasoning that seemed to flow naturally. But when the judges went looking for the actual decisions, they found nothing. No court had decided those cases. There was no record in the ordinary legal databases. The opinions existed only in the output of an AI program, which had woven them together out of patterns in its training data. Justice Valenzuela’s description of them as “hallucinations” was precise. The software was not lying in a human sense; it was generating text in the way it had been trained to generate text, and the product happened to be false. The attorneys who filed the brief apparently did not catch the problem before sending it to the court. The 4th Court of Appeals decided not to dismiss the appeal, which means the mother’s case could still move forward. But the court handed down a warning to the attorney and the law firm, making clear that future filings would be examined with great suspicion. It was a sanction dressed in polite legal language. More importantly, it was a signal to every lawyer who uses AI that courts are not interested in hearing that “the computer made me do it.” The lawyer who signs a brief takes responsibility for every word in it, including the words that came from a machine. In the professional world, there is no distinction between an intentional lie and an unchecked AI hallucination. Both undermine the integrity of the legal process. For many lawyers, this was the first time they had watched a colleague get in trouble for AI use. It forced them to ask whether their own research habits were safe. A lawyer who relies on a tool’s summary rather than reading the full text of a case is essentially taking the same risk. The difference is that a human error is easier to spot because it usually has a pattern; AI errors are often indistinguishable from correct results.
Legal experts say this should not come as a surprise. Mason Clark, an assistant professor of law at St. Mary’s University School of Law, has followed the spread of AI with a mix of fascination and concern. When asked what he sees in the legal field, his answer was direct. “Unfortunately, most of the time when I’m asked this question, what are we seeing in the legal field, we’re seeing mistakes.” That may sound harsh, but Clark explains that AI makes mistakes look polished. It has been trained on massive amounts of text, including legal opinions, and it has learned to imitate the style of legal writing. The result is a tool that can invent a case and make it sound exactly like real precedent. It can easily fool someone because it makes false information sound legal and sound good. Clark’s own words were even more blunt: “It can easily fool someone to making it sound legal, making it sounds good, and it’s really completely worthless.” In other words, the very qualities that make AI seem impressive—fluency, confidence, and speed—are what make it most dangerous in the courtroom. A human lawyer who cannot find a case to support an argument knows to stop and search again. An AI program rarely stops. It simply produces something that fills the gap. The human brain is wired to trust fluent language, and legal writing is full of formal cues that signal authority. A fabricated opinion can check all the boxes and still be nothing but a phantom. Clark’s point is not that AI should be banned. It is that no one should mistake its output for the final product. The machine is a starting point, not an end point. The lawyer is the person who must verify, understand, and stand behind every word. Law schools and bar associations are beginning to teach this lesson, but the adoption of AI is moving faster than the education. Many lawyers have not been trained in how AI works or how to use it responsibly. Others do not even realize that the tools they already use are powered by AI. That gap between usage and understanding is where hallucinations thrive.
The risks are even more serious because the number of lawyers using AI is rising fast. A recent survey by the State Bar of Texas found that AI use among Texas attorneys more than doubled, jumping from 30% to 62% in just one year. That is not a small shift. It represents a fundamental change in the way legal work is being done. The pressures behind the shift are easy to understand. Clients demand efficient billing. Firms compete on speed. Courts are overloaded. AI can scan thousands of pages, summarize depositions, spot patterns in documents, and generate first drafts that once took hours. For solo practitioners and small firms, it can level the playing field against larger competitors. But the same efficiency carries a hidden price. When a lawyer hands over part of the thinking process to a machine, the lawyer must still take full responsibility for the result. The San Antonio case is the clearest example: the appeal was not derailed because the legal argument was weak. It was derailed because someone trusted a machine to know something that cannot be known from a pattern of words. The survey did not ask whether AI users were more likely to face discipline, but the number of court opinions mentioning AI hallucinations is climbing. Some courts have already sanctioned lawyers. Others have ordered them to explain why they should not face disciplinary proceedings. The phrase “ghost law” has entered the legal vocabulary to describe fabricated cases and statutes that exist only in AI-generated text. In a profession built on precedent, ghosts are dangerous. Yet the answer is not to reject AI entirely. The answer is to use it the way a good pilot uses an autopilot: with hands near the controls, constant monitoring, and an unwavering willingness to take over at the first sign of trouble. The bar survey also hints at a generational shift. Younger attorneys are more likely to have grown up with technology and are more comfortable trusting it. But comfort is not the same as competence. Every attorney, regardless of age, is bound by the same ethical rules. They all take the same oath. They must all exercise the same judgment. The fact that a tool is digital does not change the standard of care. If anything, it raises it, because a lawyer who uses a tool without understanding its limitations is making a voluntary choice to fly blind.
The legal system is starting to respond. This summer, the Texas Supreme Court issued new rules addressing the use of artificial intelligence. The rules take effect Oct. 1 and are designed to provide clear ethical guardrails for a rapidly changing profession. They do not ban AI, and they could not, because the technology is already embedded in everyday tools. Instead, they focus on accountability. Attorneys are reminded that the duty to be truthful and candid with the court belongs to the lawyer, not to the software. Every citation, every factual assertion, and every legal argument must be checked and supported by trustworthy sources. The new rules also highlight the importance of client confidentiality. Many AI tools store the information users enter, and some use it to train future models. A lawyer who uploads privileged documents into an unsecured platform may be breaching ethical duties even if the AI produces a perfect result. The Texas Supreme Court is not trying to slow down progress; it is trying to make sure progress does not come at the expense of justice. The rules are part of a broader trend. Courts across the country have adopted local rules about AI in filings, with some requiring lawyers to disclose when they have used AI. Others have issued standing orders requiring a certification that all citations are genuine. The shift has been remarkably quick, especially for an institution that usually moves slowly and carefully. Judicial discomfort with fabricated cases has turned into formal rulemaking, and the San Antonio case played a part in that evolution. It showed that the problem is real, immediate, and not confined to a few careless attorneys. The rules do not guarantee perfect behavior, but they do something almost as important: they tell every lawyer and every judge that AI is not a magic wand. It is a tool, and tools can be used well or used badly. The rules effective Oct. 1 arrive just months after the survey showed 62% usage. That is not a coincidence. The court appears to be acting quickly to avoid a wave of incidents. It is also giving lawyers a chance to build good habits before the technology becomes even more powerful. The next generation of AI will be able to write entire briefs with almost no human input. If lawyers do not learn discipline now, they will be overwhelmed later.
At its core, the San Antonio case is a very human story about trust, pressure, and responsibility. The mother who sued the district wanted a fair hearing, and she was entitled to one. Her lawyer, presumably, wanted to help. But somewhere in the rush of legal work, a machine’s invention slipped into the middle of a serious legal filing. The result was an embarrassing detour that drained time, energy, and trust from a case that should have been about a student and a school. The 4th Court of Appeals did not dismiss the case, but the warning it issued is a reminder that the law is ultimately a human institution. No algorithm can walk into a courtroom and explain why a client deserves justice. No AI can look a worried parent in the eye and promise to do the best possible job. Those things require human beings. They require judgment, empathy, and the willingness to be accountable for every word that is submitted to a court. AI can help people do those things faster and more efficiently. It can organize facts, identify patterns, and suggest ways to frame an argument. But it cannot tell the difference between a real precedent and a dream. It cannot know when a case is too important to risk a shortcut. The phrase “I didn’t check it because the machine said so” must never become an acceptable excuse, in law or in life. The professor’s warning stands as a simple truth: AI can sound legal, sound good, and still be completely worthless. In the years ahead, everyone will have opportunities to rely on artificial intelligence. The ones who thrive will not be those who trust it the most. They will be those who trust it least—but use it wisely. They will ask the questions that matter: Does this check out? Do I understand it? Would I defend it in court? The San Antonio case is not just a warning to lawyers. It is a warning to all of us, to hold on to human judgment in an age of machine confidence. As AI gets better at sounding certain, the value of a careful, questioning human mind will only grow. And perhaps that is the real lesson. Technology does not have to be perfect for us to use it, but we do have to be honest about its limits. The moment we forget that is the moment we become the unwitting authors of someone else’s hallucination.