Large language models have been used for cheating in schools and the spread of disinformation in news reports. Now, they are creeping into courtrooms, exacerbating the false litigation faced by judges in heavy caseloads and posing new risks to an already stretched judicial system.
A recent report from Ars Technica details a Georgia Court of Appeals ruling that highlights a growing risk facing the U.S. judicial system: AI-generated hallucinations creeping into court documents and even influencing judicial decisions. In the divorce dispute, the husband’s lawyers submitted a draft order filled with references to non-existent cases — cases that were likely fabricated by generative AI tools like ChatGPT. The trial court signed the document and subsequently ruled in favor of the husband.
It was not until the wife appealed that the forged subpoenas came to light. An appeals panel led by Judge Jeff Watkins reversed the order, noting that the bogus cases undermined the court's ability to review the verdict. Watkins was outspoken, saying the subpoenas could be an AI-generated hallucination. The court fined the husband's attorney $2,500.
This may sound like an isolated case, but in February of this year, a lawyer was fined $15,000 in a similar situation. Legal experts warned it could be a harbinger of things to come. Generative AI tools are known to easily falsify information with convincing confidence—a behavior known as “hallucination.” As artificial intelligence becomes increasingly accessible to overwhelmed lawyers and self-represented litigants, experts say, judges will increasingly be faced with litigation documents filled with bogus cases, fake precedents and confusing legal reasoning masquerading as legitimate.
This further exacerbates the problem in a justice system that is already stretched thin. In many jurisdictions, judges often approve attorney-drafted judgments without a second thought. However, the use of artificial intelligence exacerbates this risk.
Court of Appeal opinion on false legal citations
“I can imagine that this might be the case in situations where trial judges are dealing with a lot of cases,” said legal scholar John Browning, a former Texas Court of Appeals judge who has written extensively about the ethics of artificial intelligence in law.
Browning told Ars Technica that he thinks it's "highly likely" this type of error will become more common. He and other experts warn that courts, especially at the grassroots level, are not ready to deal with such AI-driven haphazard practices. Currently only two states, Michigan and West Virginia, require judges to maintain basic “technical competency” when it comes to artificial intelligence. Some judges have banned the use of AI-generated documents entirely or mandated disclosure of AI use, but these policies are incomplete, inconsistent, and difficult to enforce due to the sheer volume of cases.
At the same time, AI-generated documents are not always obvious. Large language models often concoct realistic-sounding case names, specious citations, and formal-sounding legal terms. Browning said there are tell-tale signs a judge can look out for: the wrong court reporter, a placeholder case number like "123456," or stilted, formulaic language. However, as AI tools become more sophisticated, these clues may gradually disappear.
Researchers such as Peter Henderson of Princeton University’s Polaris Laboratory are developing tools to track the impact of artificial intelligence on court documents and are advocating for an open legal case library to simplify verification. Others have proposed novel solutions, such as a "bounty system" that rewards those who spot false cases before they are caught.
For now, the Georgia divorce case is a warning — not just about careless lawyers, but also about the potential for court systems to be overwhelmed and unable to track the use of artificial intelligence in every legal document. As Judge Watkins warned, if AI-generated hallucinations continue to creep unchecked into court records, they could undermine confidence in the justice system itself.