As a lawyer, one of the most terrifying artifacts of our era is Damien Charlotin’s database of AI Hallucination Cases, cataloguing instances where lawyers and pro se litigants have been called out for using faulty AI-generated material in briefs or other documents.
I’ve written previously about the depredations the judiciary can expect from the widespread adoption of AI. But Charlotin records new incidents as they happen. (And of course, just the ones that have been detected. We can suppose that many—if not most—are not.)
If you thought that incidents of hallucinated legal citations would be reported two or three times, and then everyone would have gotten the message—nah. If anything, these incidents appear to be increasing in frequency with the popularity of LLMs.
For those just arriving: LLMs are not truth machines. Rather, LLMs assign numbers to words and then assemble those numbers into sequences that we, the human operators, are meant to find convincing, after those numbers are converted back into the corresponding English words. But LLMs don’t understand English. LLMs don’t even know what language is. LLMs cannot intentionally make true statements, as that would require them to be able to distinguish true from false. Rather, an LLM generates output, which often contains sentences that happen to be true, in the same way the broken clock of proverb happens to be right twice a day. Though this comparison is a little unfair to broken clocks, whose failure modes are well understood. Not so for LLMs.
“But humans make mistakes too.” The antidote to a human making a mistake is another human who is better informed or more diligent. We have those. What’s the antidote to a horde of machines that generate material that doesn’t even try to be true (aka bullshit), at a cost that will tend toward zero, at a volume that will tend toward 100% of all textual material that exists?
We’ll get back to that.
I have predicted that the AI that comes to dominate us—which is my idiosyncratic definition of AGI, or artificial general intelligence—will be emergent, simple, brutally stupid, and consequential but not agentic. That last part means that AGI will have effects in the real world without AGI having to “want” them to happen.
A perfect example arrived this week: a sheet-music scanning service called Soundslice found that a large number of visitors were uploading a certain type of silly music notation that the service didn’t support. Why? Because a certain commercial LLM was telling large numbers of people that Soundslice supported this silly notation. So what did Soundslice do? Not wanting to disappoint these people, it implemented the feature. Life used to imitate art. Now it will imitate LLMs. In that sense, the risk created by an LLM arises not from what it wants, because it has no agency. It wants nothing. But humans do. Humans want the words spuming out of the LLM to be true. And humans will behave as if they are.
Much discourse on AI risk seems focused on what we might call agentic risks: how we make sure AI systems tend to produce output that supports what we want, and tend to avoid other kinds of output—a project known as alignment. Because alignment is difficult, an AI catastrophe arising from failure of alignment is more likely than one arising from an AI going rogue—or even more outlandishly, “sentient”. Outlandish because an AI will never need sophisticated measures to dominate us, because it can passively exert unlimited pressure, like a huge rock sitting on the chest of humanity.
How easily can alignment go wrong? Here’s a tiny yet terrifying anecdote from Karen Hao’s excellent new book Empire of AI. Below, “RLHF process” refers to “reinforcement learning from human feedback”, one technique for aligning an LLM according to human ratings of its output:
[L]ate one night, a researcher [at OpenAI] made an update that included a single typo in his code before leaving the RLHF process to run overnight. That typo was an important one: It was a a minus sign flipped to a plus sign that made the RLHF process work in reverse, pushing GPT-2 to generate more offensive content instead of less. By the next morning, the typo had wreaked its havoc, and GPT-2 was completing every single prompt with extremely lewd and sexually explicit language.
Still, a key premise of alignment is that these flaws of reasoning are internal to the AI system, and there is some way we can—even theoretically—control them.
But no AI system exists in isolation. It interacts with humans. Humans who are all too happy to trust and operationalize the hallucinations of LLMs, thereby magnifying their effects. Multiplied enough times, in enough places, and we can foresee increasing risk of what I’ll call a zero-agency catastrophe—an AI catastrophe brought about not by malevolent AI, or even failure of alignment, but humans imputing the same level of trust to LLMs that they would to other humans, leading to wretched consequences.
For that reason, law is much more vulnerable to erosion by LLMs than the sheet-music service described earlier. That company at least had a choice whether to update their product to match the words coming out of the LLM. In law, by contrast, the words on the page are the product, as the database of AI Hallucination Cases reminds us.
This leads to what I see as two key vulnerabilities:
Law itself. Above, I asked what antidote we have to the horde of bullshit-generating machines we’re putting in the hands of lawyers at basically zero cost. Did you think of one? No? Then maybe you will join me in believing that LLM output, as it grows exponentially, will gum up the law in ways small and large, leaving behind toxic, untrustworthy slop in the form of bullshit-filled documents.
Not in elite products like U.S. Supreme Court briefs, which will always be tended to by highly diligent humans. Rather, the slop will increase as you get closer to the courts of first resort, closer to the huge number of criminal, family, landlord-tenant, juvenile, and probate cases that affect the liberty and property rights of millions. In short, the costs of the slop will fall on those least able to bear it. Not for the first time.
Legal practice. Because they currently lose gargantuan sums, AI companies will be selling hard to anyone who still makes money on writing. As I’ve noted elsewhere, the legal industry is the biggest publishing industry in the US. I expect that the transition to AI within law offices will proceed as Hemingway described the descent into bankruptcy: gradually and then suddenly. The “gradually” is happening now. The “suddenly” will start when the first huge law firms start laying off document reviewers and support staff, and inhibiting their hiring of associates. This will lead to cost-structure advantages that will in part be passed through to clients. Competitor firms will have to match these cuts or risk having these AI-powered rivals grow even larger and crowd them out.
I’m no labor economist, but it hasn’t stopped me before: I’m not sure big-firm partners have fully penciled out the lifestyle ramifications of this transition. As I understand it, big-firm partners are paid more than they bill directly. This is possible only because lower in the firm hierarchy, there are larger groups of people who are billed at a surplus to what they get paid. The money flows upward. Not for the first time.
But as those lower tiers of labor are automated away by AI, the trickle-up economics will necessarily fail. By contrast, in a manufacturing context, the automation of labor is a benefit in strictly economic terms, because the same product is getting made at lower cost. Whereas in a company built around hourly billing of human labor, AI will be devastating, because it makes the product disappear.
“But law firms automated away a lot of typists with desktop computers, and that was fine.” I didn’t run a big law firm in the 1980s. But I’m skeptical that analogy is economically apt. Law firms bake many support costs into the rates of their billable staff. If those support costs can be automated, then of course it’s beneficial. Costs always want to be minimized. With AI, however, the automation replaces the billable services. So it’s a cost shrinker, but also a revenue shrinker.
Against that backdrop, I expect that a rising wave of AI tools targeted at legal practice will eventually nudge many firms to either a) consciously avoid AI, and collectively accept lower profits per partner in line with their firm’s less competitive economics; or b) accept minimum AI, and dismiss just enough equity partners to keep the profits per partner stable; or c) embrace AI, and cut deeper into the partner ranks to concentrate the returns from AI around a smaller group, thereby increasing profits per partner. You don’t need an LLM to guess which of these paths are most likely.