When Rebecca Green arrived for her first appointment with a urologist, she was met with an unexpected question: did she consent to having the consultation transcribed by artificial intelligence? She said yes without hesitation, not because she had weighed the risks, but because she didn’t want to be “that annoying patient” who slows things down or irritates a busy specialist. It was a small decision made under the quiet pressure of a vulnerable moment, and it would come back to haunt her. Months later, after undergoing surgery for kidney stones, she read the post-operative letter the specialist had sent to her general practitioner. There, buried in her own medical record, was a claim that stunned her: the AI scribe had invented a detail that she micro-dosed psychedelic mushrooms, and suggested this could explain previous bleeding around her kidneys. Rebecca Green has never taken mushrooms in her life. The fabricated inclusion felt like a nightmare—not only because it was false, but because it carried real consequences. She was receiving workers’ compensation at the time, and any mention of illegal drugs in her official health files could threaten her case, her income, and her reputation. She described feeling “gobsmacked and in tears.” More than the mistake itself, what wounded her was the obvious absence of a human check. If the doctor had read the letter carefully before sending it, she reasoned, the error would have been caught immediately. Instead, she was left to discover it herself, having trusted a system that seemed too distant, too automated, and too careless to protect her when she was at her most fragile.
AI scribes are becoming increasingly common in Australian healthcare, and for good reason. They promise to lift an enormous administrative weight from overworked doctors, allowing them to focus on patients instead of screens and reducing the burnout that has pushed many clinicians to the edge. The Royal Australian College of GPs estimated last year that around 40 percent of GPs were regularly using these tools, and many believe the real number is higher. For every benefit, though, there is a risk, and a new report from the advocacy group Digital Rights Watch has laid those risks bare. Australians, the report warns, are being exposed to unregulated AI medical scribes, sometimes without meaningful consent, and the errors these systems make can change clinical decision-making in serious ways. The report documented cases where a scribe recorded the wrong breast in a breast cancer diagnosis—a mistake with potentially catastrophic implications—and other instances where patients were falsely labeled as having epilepsy. These are not trivial mix-ups; they are distortions of a person’s medical identity, and they can follow someone for the rest of their life. Rebecca Green feels lucky that she spotted her own error, but she worries about the many patients who never look closely at their records or who lack the confidence to challenge what they find. In an age where private health insurers have also been known to demand full clinical histories from providers, an incorrect note about illegal drug use could ripple outward in ways far beyond the doctor’s office, affecting employment, insurance premiums, and even the quality of future medical care.
The way consent is handled in this new AI era is deeply inconsistent, and that is one of the report’s most alarming findings. Some doctors simply place a sign in the waiting room announcing that AI may be used, which falls far short of what privacy laws require. Others demand consent at the moment a patient books an appointment online, before they have any real chance to ask questions or understand what they are agreeing to. And in some cases, patients who decline are refused a consultation altogether—a deeply uncomfortable reversal of the patient’s right to choose what happens to their own body and data. Rebecca Green’s experience shows how easily consent can become a hollow checkbox. She said yes not out of informed enthusiasm, but out of fear of being judged difficult, or of consuming more of the doctor’s already scarce time. That is not genuine consent; it is compliance under pressure. Elizabeth Deveny, chief executive of the Consumers Health Forum of Australia, argues that consent must be enthusiastic and fully informed, and that patients should never feel intimidated about asking what the AI is being used for, who will see the transcript, and how they can correct mistakes. She says she frequently hears from people who have had to fight to get wrong records amended, often struggling to find the original source of the error and then encountering resistance when they try to fix it. Some patients, the report also notes, may become more guarded during consultations when they know AI is listening, especially about deeply sensitive topics like family violence. When a patient cannot speak freely, the entire purpose of the consultation is undermined.
Doctors themselves acknowledge that AI is not infallible, but they stress that responsibility ultimately rests with the human practitioner. Perth GP Sean Stevens has been using an AI scribe for three years and has become a vocal advocate for the technology, serving as chair of the RACGP’s Digital Health and Innovation group. He says the accuracy has improved dramatically since he started—early on, he had to correct multiple mistakes in every transcript, while today he might see one correction every third or fourth transcription. Yet he is blunt about the technology’s flaws. “The classic is, it will get the side of the body incorrect and say right when you’ve said left,” he explained, adding that drug doses and other critical details require especially close attention. The national regulator for medical practitioners, AHPRA, has made it clear that clinicians must always check all output from AI scribes to meet their professional obligations. But as Elizabeth Deveny points out, there is troubling evidence that doctors become complacent after the first few weeks of using such tools. They check carefully at first, and then gradually stop assuming the AI will get things right. That complacency is exactly what seems to have happened in Rebecca Green’s case. Dr Stevens insists that using AI does not allow him to squeeze more patients into his day, but it helps him run on time and produce more thoughtful, comprehensive referrals. Still, he says consent must be asked at every appointment, even for regular patients, because patients must know they can say no—and a doctor must respect that choice. The technology, in other words, is only as safe as the people using it, and when humans stop paying attention, patients end up paying the price.
Regulation has lagged dangerously behind this explosion of AI use. There are around a dozen AI medical scribes available in Australia, but none have been approved by the Therapeutic Goods Administration, the country’s medical device regulator. Because these products claim to be purely for transcription and summarising, they fall outside the TGA’s oversight framework. The TGA only considers a tool to be a medical device if it does more than that—for example, if it suggests a diagnosis or recommends a treatment plan. Yet some AI scribes already offer clinical suggestions, which means they may be operating in legal breach without ever having been assessed for safety. The TGA has spent the past year reviewing these tools and says its findings will be released in the coming weeks, with promises that any breaches will be acted upon. Meanwhile, patient advocates are calling for stronger safeguards. Lisa Robins, chief executive of Patients Australia, argues that all scribes should be regulated, because summarising a patient’s health information is itself a form of clinical decision-making, one that carries enormous risks when done incorrectly. Digital Rights Watch is urging the newly established AI Safety Institute to independently test these scribes and publish a list of approved tools, similar to New Zealand’s approach. In New Zealand, the national AI advisory group reviewed available products and last year granted approval for clinicians to use two AI tools: Heidi Health and iMed. There is also growing support for mandatory checks built directly into the software, so that errors are caught before they reach a patient’s record, and for regular inspections during practice accreditation, which occurs every three years. Dr Stevens, for his part, worries that heavier regulation could stifle future innovation and discourage developers from improving these tools. It is a familiar tension—between the need to protect patients and the desire to embrace technology—but for Rebecca Green, the scales have tipped decisively.
Rebecca Green’s trust in AI has been deeply splintered by what happened to her. She says she still loves the idea of AI and believes it could bring great benefits to healthcare, but she now understands that the essential ingredient is human involvement. “At the end of the day what’s happened here is more about the lack of human involvement,” she reflected. For her, the urologist’s failure to review the AI-generated letter was not just a clerical oversight; it was a failure of care, a break in the sacred relationship between doctor and patient. She said she would only consent to AI transcription again if she had an established, trusting relationship with the doctor, someone who knew her history and would notice if something seemed wrong. Her story is a warning to all of us: medical notes are more than administrative paperwork—they are the foundation of future treatment, legal evidence, and a record of who we are. When technology is allowed to write that record without careful human oversight, patients are left vulnerable in ways they never agreed to. As AI becomes further embedded in Australian healthcare, the challenge will be to remember that behind every transcript is a living person, anxious, hopeful, and deserving of care. We cannot allow the speed and convenience of automation to erode the diligence, empathy, and accountability that patients have every right to expect. If Rebecca Green’s experience teaches us anything, it is that the machine may have written the words, but only a human can truly listen.