Most people assume a hallucination is a bug waiting on a patch, and the roadmap will fix it. Then a single fabricated sentence moves past the model output and becomes a callback, a credit, a case, a lawyer.
The demo showed a 400-seat customer-support assistant at a US health plan answering a benefits question fluently, with a citation, in under two seconds. That version got the budget approved.
What the demo left out: in that deployment's first quarter, 3 in every 1,000 answers contained a made-up policy detail, logged internally and caught mostly by humans. The demo priced the answer, not the repair.
Follow a fabricated coverage detail or clause citation and it lands on a specific invoice line. You will see how to trace that chain yourself, and what to demand in writing before renewal: a measured fabrication rate, an indemnity cap, and review hours.
A Wrong Answer Becomes a Billing Event
A made-up policy sentence costs $48 the moment a real member believes it. That figure is the average repair bill at the 400-seat health plan, and it is assembled from labor nobody itemized.
Trace one. A member asks whether a specific home-care visit is covered. The assistant invents a prior-authorization waiver that does not exist, cites it in the right format, and returns an answer in under two seconds. The member schedules the visit. The claim is denied. The phone rings.
From there the touches stack in order: a callback from a service agent, an account credit for the failed claim, a case opened in the service system, and that case closing onto a monthly invoice. Five paid touches, and not one of them creates a ticket in a bug tracker. The engineering team sees a healthy model.
Severity is not what runs the bill up. At 1.2 million chats a month, 3 fabricated answers per 1,000 puts the $48 repair through thousands of cycles, every month.
| Cost line | What triggers it | Amount in the example |
|---|---|---|
| Agent callback and account credit | A wrong answer reaches a customer and is caught | About $48 per incident |
| Outside counsel and remediation plan | A fabricated coverage denial reaches a state regulator | $310,000 and seven months |
| Human review and guardrails | Every answer that needs a check or a citation | $1.9 million in year one (about 40% of the assistant's budget) |
| Vendor indemnity cap | The most the contract will ever pay out | $240,000 (12 months of fees) |
Most of the Bill Is Prevention, Not Payouts
Both of those numbers are payouts, and payouts are the smaller line. The larger, recurring spend at the health plan is the machinery that stops a fabricated clause from ever reaching a member: retrieval over a controlled corpus, output guardrails, and a staffed human review tier.
That apparatus cost $1.9 million in year one.
Set it against the total assistant budget and the shape of the thing becomes clear: roughly 40%, which makes prevention bigger than inference, the line the approval deck was actually built around. The slide that won the room priced tokens. The line that funds the error is review.
Which is why the procurement argument over liability caps is aimed at the wrong number. Payouts swing with luck; prevention is charged every year of the contract, at roughly the same rate, whether or not a single regulator ever calls.
Regulated work is the exception, and it inverts fast: one incident that becomes a regulatory matter can outweigh a full year of that prevention spend on its own.
The Billions Come From Volume, Not Disaster
Run the multiplication and the headline stops being a headline. At 1.2 million chats a month, 3 fabricated answers per 1,000 gives roughly 3,600 bad answers monthly; at $48 to repair each, about $173,000 a month, or just over $2 million a year.
All of it from a rate low enough to look like a rounding error in a pilot report.
Industry-wide totals are built by exactly this multiplication across deployments, and they are disputed – the published figures move with the assumed fabrication rate, the assumed repair cost and the assumed seat count, none of which vendors disclose uniformly.
Go to the source of any headline number and check those three inputs rather than accepting the total. A total without a stated fabrication rate is a marketing artifact.
The per-incident number stays small – $48 – and that is precisely why it survives budget review. It sits below the threshold that triggers a line item in year one, and above the threshold that compounds by year two.
Fabrication Is the Mechanism, Not a Bug
Vendors cannot patch fabrication out because the made-up clause and the correct one leave the same generation process in the same format.
Two different failures get folded into one conversation at the contract table. A retrieval failure means the source document was missing or never fetched – a plumbing problem, fixable with better indexing. Fabrication means the source was present, retrieved and cited, and the model still invented a clause, a coverage limit or a policy detail that was never in the document. The first is an engineering ticket. The second is the mechanism doing what it was built to do.
A correct citation and an invented one arrive at the same confidence, in the same case-name styling, through the same next-token prediction. No patch strips one and keeps the other, because they are the same output.
What a vendor genuinely can deliver: grounding in a controlled corpus the buyer can audit, a requirement that every claim cite a retrieved passage, and refusal when retrieval quality is low. Each of those bends the fabrication rate down – the insurer's measured rate fell after adding them – and none reaches zero. What no vendor can warranty is that an answer will never contain invented content, as seen when Claude AI wandered off task, because that guarantee would demand a generation process that is not prediction.
Which is why a 'known issues' list that mentions hallucination is a disclosure, not a remediation plan. It carries no rate, no date, no credit.
For the reader, that list is unpriced risk moving across the table to their side of the ledger – signed for at renewal, funded by their service organization, invisible on the vendor's invoice.
What vendors can fix vs. what stays with you
Genuine fixes (reduce the rate)
- Grounding in a 400-seat corpus audit: reads 3 fabrications per 1,000 answers
- Mandatory citation to a retrieved passage cuts review from 25 minutes to 4
- Refusal when retrieval quality is low – test the threshold in writing
- Fine-tuning fixes tone, not the 3-in-1,000 fabrication floor
Unfixable (carried by the buyer)
- Zero fabricated answers in a 12-month term
- Regulator letter after one wrong coverage denial: $310,000 and seven months
- Callback, credit and follow-up per escaped answer: about $48
- Indemnity caps at 12 months of fees – $240,000 against your losses
Fluent and Formatted Is What Makes It Dangerous
A reviewer reading twenty summaries in an afternoon cannot spot a pattern that appears eleven times in twenty thousand. At a 900-person software firm running contract review, those eleven summaries carried clause citations to cases that do not exist, buried in pages of otherwise accurate paraphrase.
Eleven sounds like nothing. That is the problem.
Each one a reviewer did flag cost 25 minutes of lawyer time to check and correct, billed at $340 an hour, so roughly $142 per item before the drafter had to redo anything. And the flagged ones are the minority — the ones that got through outnumbered them, because nothing in the document announced itself as wrong.
The fake citation looks exactly like a real one: proper case-name italics, a docket number in the right format, a confident sentence introducing it. It sits in a paragraph whose other four sentences are correct, which is what trains the eye to skim.
So the bill that recurs is not the fabrication. Producing a wrong sentence costs the buyer nothing incremental — it is output. Checking the sentence costs $340 an hour, and once trust in the tool is gone, that check runs on every summary that ships, not just the bad ones. A lower fabrication rate does not lower this bill proportionally: the review is sized to what a miss would cost when it reaches a client, not to how often misses occur.
💡 PRO TIP
Size the review, not the error
When you build the business case, multiply the fabrication rate by the review hours per output, not just by the repair cost. At $340 an hour, a 25-minute review of one flagged summary is $142. A rate that looks small in a pilot report can still staff a full-time reviewer across 20,000 outputs.
Regulated Industries Pay the Highest Price
A wrong answer that gets refunded in five minutes and a wrong answer that gets filed with a regulator are the same error with two very different invoices. At the health plan, one fabricated coverage denial left the support queue entirely because a denial is a document: issued to a member, logged, reportable and auditable months later. The error rode the compliance system, not the service desk.
The remedy was paperwork, not refunds — seven months of remediation and $310,000 in outside counsel and staff time, from a single answer.
High-volume, low-stakes work stays cheap. Draft marketing copy, internal search, ticket tagging: a wrong line is caught or shrugged off, corrected in minutes, and never appears in a file anyone audits.
The test to run on your own deployment takes one question. In your industry, when an answer is wrong, is it refunded or filed? Refunded puts you on the $48 callback-and-credit chain. Filed starts the seven-month clock, and the count on the invoice is hours, not the price of a query.
Same model, same measured error rate, same contract terms — two cost profiles that differ by an order of magnitude depending on which side of that line your output lands on. The rate tells you how often; the filing tells you what each one is worth.
Vendor Indemnities Stop Well Short of the Loss
The vendor pays for a narrow slice, and the slice has a ceiling. In the example contract that ceiling is 12 months of fees: $240,000. One fabricated coverage denial that reached a state regulator cost $310,000 in outside counsel and a seven-month remediation plan. The cap is smaller than the single incident before you count the $48 repair chain or the $1.9 million in prevention spend.
What the indemnity actually covers is intellectual-property infringement, meaning the vendor's model reproduced someone else's protected text. It does not cover accuracy. It does not cover whether an answer is fit to send. Read the clause in the contract and you will find the words are about copyright, not correctness.
Buyers read it as covering correctness anyway, because the section is titled Indemnification and sits near the liability language. That misread is expensive.
Three levers move real money, and none of them is a cap increase. First, an accuracy SLA tied to a measured fabrication rate on the buyer's own corpus. Second, an error-rate warranty that names the sample size, the sampling method and the definition of a fabrication. Third, remediation credit that pays for review hours and counsel rather than refunding subscription fees you were going to spend anyway.
Regulatory exposure stays with you regardless of the cap, because a state regulator does not fine the model vendor for a denial issued under your license and your name.
Convert the cap to dollars and set it beside the cost of one filed incident in your own industry. If the cap is smaller, the indemnity is a rounding error dressed as risk transfer.
Buyers Budget for Tokens, Then Pay for Reviewers
Nobody sees it coming because the approval deck prices the only things that are metered. Cost per query, cost per seat, cost per month: clean inputs, easy to model, and they win the room every time. Review hours are not metered, so they do not make the slide.
That gap is where the ~40% prevention figure lands, and it lands after launch. It arrives as an operations surprise, on a different leader's P&L, in a quarter after the business case was signed off. The people who approved the budget are not in the room when the reviewer headcount request comes through.
The sign-off split makes the blind spot structural. The AI team ships the assistant and owns the model relationship. The service organization staffs the reviewers and absorbs the volume. Nobody owns the combined number, so nobody is positioned to flag it.
The budget line to add before launch is reviewer hours per 1,000 outputs, priced at the loaded hourly rate of the actual person doing the review, carried for the full contract term rather than the pilot. A pilot that runs 50,000 outputs a month hides this line completely.
If you approved the budget, the question before go-live is not what a query costs. It is which cost center pays the reviewer for the 3-in-1,000 and for the near-misses that never get logged at all.
The error-to-invoice chain in five paid touches
Figures from the example deployment described in this article.
Guardrails Cut Errors and Raise the Bill
Three controls genuinely move the fabrication rate, and every one of them adds a line to the budget. Grounding in a controlled corpus the buyer can audit. Required citation to a retrieved passage on every factual claim. Refusal when retrieval quality or confidence is low.
Each control drags infrastructure behind it: retrieval pipelines, citation enforcement, refusal handling, and the review tier that watches all three. At 400 seats and 1.2 million chats a month, that stack is the $1.9 million figure from earlier. Controls are a purchase, not a fix.
In the health plan's deployment the measured fabrication rate fell once these were in place. The size of the drop is deployment-specific and not a published constant, which means your own measured rate is the only number worth contracting against.
Skip fine-tuning as a factual fix. It changes tone and format far more than grounding. Skip 'please do not hallucinate' system prompts, which do nothing measurable. Skip vendor demos entirely: a curated sample has no denominator, so it cannot tell you a rate.
The negotiation narrows to three questions. Which controls the vendor must supply, which the buyer must staff, and what measured rate triggers the remediation credit. Get all three in the signature memo.
Guardrails stack — at a glance
📚 Grounding
Auditable controlled corpus
🔗 Citations
Required on every claim
🛑 Refusal
When retrieval is weak
💵 Year-one cost
$1.9M at 400 seats
📉 Effect
Rate falls, never zero
Buyer's line
The stack is a recurring operating expense, not a one-time integration fee.
Three Questions to Ask Before You Sign
Take a measured fabrication rate on your own corpus from the vendor's solutions engineer before a dollar moves. Ask for the number on your policies, your contracts, your document set, not a benchmark score from a generic evaluation set. Require three attachments in writing: the sample size, the working definition of a fabrication, and the calendar date the measurement was run. A rate quoted without the denominator is a slide.
Then bring the indemnity to legal and price the cap in dollars. Twelve months of fees reads as generous in a term sheet and lands at $240,000; the single fabricated coverage denial that reached a state regulator cost $310,000 in outside counsel and remediation before any repair chain. Put that arithmetic in the signature memo, line by line, next to the fees.
The third person sits inside your own building. Your service lead needs to state who pays for review hours, how many hours per 1,000 outputs, and which cost center carries that line for the full contract term. Silence here is a staffing gap that surfaces in a quarter after sign-off.
A vendor who declines to produce the first number on your own data has already answered. The rate is unknown, and the discovery work that would have revealed it is being funded on your budget.
Frequently Asked Questions
What exactly counts as an AI hallucination in an enterprise deployment?
A hallucination is a confident, fluent output that contains invented content — a policy detail, a clause citation, a coverage limit — even when the source material is present and was retrieved. It is distinct from a retrieval failure, where the source document is missing altogether. The distinction matters because retrieval failures have engineering fixes; fabrication is the generation process itself.
How do you measure a fabrication rate on your own data?
Take a statistically meaningful sample of live outputs — a few thousand is enough to see a pattern — and have a domain expert mark each one for invented content against the source corpus. Record the definition of a fabrication, the sample size, and the date. The vendor should supply this on your data, not a benchmark, before you sign. If they will not, treat the rate as unknown and budget the review accordingly.
Does a lower fabrication rate reduce the cost of human review?
Not proportionally. Review is sized to the consequence of a miss, not the frequency of an error. At a 900-person software firm, 11 fabricated citations across 20,000 summaries still cost 25 minutes of lawyer re-review at $340 an hour per flagged item. The review layer becomes a permanent operating expense once trust in the system is gone.
What should a vendor indemnity actually cover?
Standard indemnities cover intellectual-property infringement, not accuracy or fitness of an answer. What you want is an accuracy SLA tied to a measured fabrication rate, an error-rate warranty with a defined sample and method, and remediation credit that pays review hours and counsel rather than refunding fees. Compute the cap in dollars and compare it to the cost of one filed incident in your industry.
Why can't fine-tuning fix hallucinations?
Fine-tuning changes tone, format and style more than factual grounding. The model still predicts plausible next text, so a correct citation and an invented one arrive through the same process at the same confidence. Grounding in a controlled corpus, mandatory citations to retrieved passages, and refusal when retrieval quality is low are the controls that actually move the rate — and they reduce it, never to zero.
Who inside the buying organization should see the measured fabrication rate?
The service lead who staffs the reviewers, the legal team that signs the indemnity, and the finance owner of the budget that carries the prevention and repair costs. The AI team owns the model relationship but rarely owns the combined number. Getting all three in the same room before signature is the point of the exercise.
Measure Fabrication Rate Before You Sign
The first move is not a review, a pilot extension or a legal escalation. It is a written demand, before signature or renewal, for the vendor's measured fabrication rate on a sample of your own data, with the sample size and the fabrication definition attached.
That number must be seen by two people inside your organization before anyone signs: the service lead who will staff the review hours and carry the volume, and the finance owner who approved the original budget – the two cost centers the rate lands on.
The demo that opened this was real: the assistant answering fluently in under two seconds. So is the 3-in-1,000 behind it. The demo priced the answer; the contract you are about to sign prices everything else.
An unpriced fabrication rate is not a technical unknown. It is a budget you have agreed to fund without seeing the amount. Demand it in writing before you sign.
Before you sign, ask for the rate on your data
Three questions, one number, and a budget line. Put the measured fabrication rate on your own corpus in writing — and get it in front of the service lead, legal and finance before anyone picks up a pen.