Imagine a scientist staring at a genomic database, trying to decide whether a newly reported gene-disease association is real. In the past, the main worry might have been a flawed experiment or a sloppy statistical test. Today, a new worry sits underneath those familiar problems: the text, data, and images in front of the researcher may have been shaped by generative artificial intelligence with no incentive to be true. A new governance analysis argues that this is not a story about a few bad actors who abuse AI, but about the entire system in which science is produced. Writing in Frontiers in Genetics, Dov Greenbaum of Yale University, Reichman University, and Berkeley Law describes how AI systems embedded in scientific publishing are being pushed by structural rewards to optimize for what gets published, cited, and funded, rather than for what is empirically real. The result can be outputs that look scientifically flawless but have no connection to the physical world. Greenbaum is careful to say he is defending AI, not attacking it. In genomics, AI has already helped shorten rare-disease diagnostic odysseys, rank candidate variants, and identify drug-repurposing opportunities where commercial incentives are absent. Human analytical capacity has simply been overwhelmed by population-scale data, polygenic scores, and interactions across thousands of gene loci. The problem, he argues, is that a tool with such extraordinary promise is being deployed inside an incentive structure that already rewards publishability over validity, and that this mismatch demands governance proportional to the stakes.
The first kind of integrity failure is the quiet contamination of the scientific record by AI hallucination. When a lawyer uses AI and gets a fake legal citation, the error can be checked against a closed, indexed corpus of court decisions. But when an AI trained on genomics generates a variant-phenotype association, a protein interaction, or an effect size that never existed, the fabrication inherits the surface features of real biology. It may carry plausible p-values, reasonable confidence intervals, and a biologically coherent mechanism, because the model learned those patterns from actual papers. This is already visible at scale. A Nature analysis found that tens of thousands of publications in 2025 may contain invalid AI-generated references. One study of 13 large language models reported hallucination rates as low as 14 percent and as high as 95 percent depending on the model and task. Another analysis of 2.2 million citations found that the rate of invalid citations rose by 80.9 percent in 2025. The tragedy is that these are not always deliberate lies. They are statistical artifacts of a language model doing what language models do: producing plausible text. But in genomics, a fabricated claim is not just a nuisance. It enters databases that other researchers use as trusted infrastructure, and from there it shapes future experiments, clinical interpretations, and therapeutic decisions. A hallucinated association that looks biologically real is much harder to catch than a hallucinated court case, because the universe of possible biological claims is open-ended and impossible to index.
The second layer of failure is deliberate fraud made cheaper and more scalable. Producing a convincing Western blot used to require a skilled forger with access to lab equipment and a deep understanding of how real data should look. Now it requires a clever prompt. The same is true for flow cytometry plots, microscopy images, and histology slides. This means AI has industrialized the paper mill, allowing synthetic manuscripts, fabricated datasets, and coherent but false peer reviews to be produced at volumes that human fraud networks could never match. Tools designed to catch copy-paste artifacts and suspiciously clean data are increasingly powerless against AI-generated images that are trained to include realistic noise, random variation, and plausible imperfections. But the most subtle failure may be the third one: epistemic blurring. Even when AI-assisted research is accurate, it can quietly dissolve the conditions that make science self-correcting. Analytical pathways become impossible to reconstruct because the AI system cannot explain exactly why it made a particular choice. The chain of custody that citation historically provided, the ability to trace a claim back to the actual work that produced it, begins to weaken. A fabricated claim can be retracted once someone identifies it. But lost provenance and attribution are much harder to repair. Once the connection between a finding and its original evidence is broken, the scientific community has lost more than one bad paper; it has lost the ability to know which parts of the literature are trustworthy at all.
Underneath all three failures is what Greenbaum calls the reward function problem. Every AI system optimizes for its embedded objective. In scientific systems, the relevant rewards are not truth but the indicators that historically correlate with scientific success: publication, citation, novelty, grant funding, and downstream impact. If you train an AI co-scientist on successful scientific outcomes, it does not learn what is true. It learns the statistical and structural features that predict acceptance, citation, and funding. Human scientists have a partial corrective mechanism because they carry an internal conception of truth that is independent of institutional incentives. They know, at least in their quieter moments, that something can be publishable and still wrong. AI systems have no such intrinsic brake unless their objectives or architectures include one. The danger is not any single false finding. The danger is that the ratio of empirically grounded claims to empirically untethered claims shifts so dramatically that the cost of telling them apart becomes too high for the whole system to bear. The feedback loop makes this worse. AI outputs optimized for publication enter databases and repositories, and the next generation of AI systems trains on that literature. Those systems learn the features of successful science with increasing precision, including the features of polluted science. In genetics, this is especially severe because the field depends on cumulative infrastructure. Once contaminated outputs enter a resource like ClinVar or the GWAS Catalog, downstream annotation pipelines treat them as ground truth. Recurrence in the literature begins to substitute for independent confirmation, so a claim that is cited three times becomes not three times more likely to be true but three times harder to retract. The article points to sobering evidence: among 166 gene-disease associations studied three or more times, only six replicated consistently, and more than 70 percent of 1,576 surveyed researchers reported failing to reproduce another scientist’s experiments. A hallucinated association does not enter a clean system. It enters a literature already saturated with non-replication, fragile claims, and hidden weaknesses.
The analysis then asks whether science’s existing self-governance institutions can handle this threat, and finds each one structurally mismatched to the problem. Granting agencies can shape research conditions, but they cannot audit every review for hallucinated citations. Journals face peer review under siege. An early systematic analysis found that only 24 percent of the 100 largest academic publishers had issued generative-AI guidance. AI-detection tools remain fragile and easily evaded. Prompt injection, the embedding of hidden instructions inside a manuscript’s text or metadata, can manipulate an AI reviewer’s judgment without the human editor ever realizing it. Universities and professional societies govern through norms rather than enforcement, and norms move on timescales that generative AI does not respect. Most importantly, there is no external regulator for truth claims. No licensing body governs scientists as knowledge producers. No pre-market approval is required for a published finding. This is why Greenbaum proposes a different approach: embed governance directly into the infrastructure through which genomic knowledge flows. Near-term measures include system-level retractability, so that when a model version is discredited, every output built on that version is flagged for review rather than managed article by article. Public credibility ledgers would make an AI system’s auditable performance history visible to journals, funders, and databases. Auditable computational lab notebooks would allow users to ask why a system made a particular variant classification or drug-target prediction. Conflict-of-interest disclosures would cover training data provenance and commercial incentives. Trust-weighted indexing would incorporate credibility signals into search ranking, so that articles with auditable provenance appear earlier. Reproducibility metrics would sit beside citation counts, so that a paper replicated three times is distinguished from a paper cited three hundred times but never directly reproduced. Above all, the article argues, replication itself should be treated as first-class scholarship. Trust cannot be policed into existence; it must be rewarded into existence.
Medium-term and continuing measures would go further. The article proposes sanctionability and exclusion mechanisms for people and organizations that knowingly abuse AI. It raises the possibility of bonding or insurance requirements for high-risk research AI systems, so that those who deploy them have real skin in the game. It suggests periodic revalidation of research AI systems, much like the recertification required for medical devices or financial models. It also proposes controlled epistemic stress-testing, in which a consortium of curators, editors, and integrity researchers deliberately introduces fabricated but plausible test cases into closed, controlled partitions of the scientific ecosystem. The goal would be to measure how quickly contamination propagates and which systems are most vulnerable. Greenbaum is emphatic that such test cases must never enter unrestricted scholarly databases, clinical systems, or public training corpora. They can only exist in environments designed to prevent uncontrolled downstream spread. Most of this agenda requires no new technology and no new legislation. It requires institutional will and funder coordination, implemented through the authority that journals, funders, database curators, indexing services, and professional societies already possess. The window for preventive action is finite. Once hallucinated findings enter shared databases, get cited by later articles, and are ingested as training data by the next generation of AI systems, contamination becomes recursive and self-reinforcing. The scientific record is not only for scientists. It underpins medicine, law, public health, and policy. Unreliable claims do not stay inside academic journals; they propagate into clinical decisions, regulatory frameworks, and public policy. The immediate task is not to eliminate AI error, which is impossible. The task is to prevent opaque error from becoming durable scientific infrastructure, and to realign incentives so that trustworthiness is competitively rewarded before the gap between plausibility and verification becomes unbridgeable. The choice is not whether genomics will use AI. That choice is already made. The question is whether the systems built around AI will reward truth, or only the appearance of truth.