While researching this article, we asked an AI tool to summarise a 43-page court order. The summary came back tidy and confident. It also had the date wrong by six months, merged two lawyers into one person who does not exist, and named a case the order never mentions.
Each error took a single search of the original document to catch. The hard part was deciding to look. This is the routine we use to make that decision quickly, and to check the claims that matter without spending an afternoon on it.
The routine at a glance
Budget about five minutes for a typical answer of a few hundred words. That figure is a working guide, not a measurement; a long report will take longer.
- Mark the checkable claims (about 1 minute). Names, numbers, dates, quotes, citations, and anything phrased as “studies show”.
- Pick the ones that could hurt you (30 seconds). The claims you will repeat, act on, or put your name to come first.
- Go to the original (about 2 minutes). Not another summary, and not the AI again.
- Confirm the source says what the AI says (about 1 minute). Existence is not support.
- Decide (30 seconds). Keep it, fix it, cut it, or flag it as unverified.
Steps 1 and 3 are the core of the lateral reading method that university libraries teach, where you break an answer into searchable claims and check them outside the tool. Step 3 is also the last of Mike Caulfield’s SIFT moves: trace claims, quotes and media back to their original context. What we add is the triage in step 2 and the insistence on step 4, because those are where AI output most often slips past people.
Step 1: mark the checkable claims
Read the answer once with one question in mind: what here could be looked up? Underline:
- Specific numbers: percentages, prices, limits, counts, dates.
- Proper nouns: people, organisations, products, court cases, papers.
- Quotes, especially neat ones attributed to famous people.
- Citations: anything with an author, a title, a journal or a case number.
- Unattributed authority: “research shows”, “experts agree”, “it is well established”.
Everything else (reasoning, structure, suggestions) is not a fact-check job. Judge it on its merits.
Step 2: pick the ones that could hurt you
You will rarely have time to check everything, so rank. Check first:
- claims you will repeat to someone else, in an email, a report or a post;
- claims you will act on: a price, a deadline, a legal or medical point, a config value;
- claims that are suspiciously specific, like a statistic to one decimal place with no source;
- anything about what is current: prices, plan limits, “the latest version”, who holds a role.
A vague claim that does no work in your document can wait. A precise one you are about to send cannot.
Step 3: go to the original
The right place depends on what kind of claim it is.
| Claim | Where to check | Fast test |
|---|---|---|
| Research paper | Google Scholar or Crossref search, then the paper itself | Search the exact title in quotes. No match usually means no paper |
| Court case | The court record, e.g. via CourtListener | Case name plus court plus year |
| Price, limit or feature | The vendor’s own pricing page or documentation | Check the page’s date, not just the number |
| Statistic | The original report, not the article quoting it | Find the actual table or sentence |
| Quote | The original transcript, video or article | Search the exact phrase in quotes and check who said it, and when |
| News event | The primary announcement, or two independent outlets with named reporters | Match the date |
VCU’s library guide makes a point worth repeating for quotes: online quotation lists often repeat misattributions, so a quote appearing on ten websites is not confirmation.
Step 4: confirm the source says what the AI says
This is the step people skip. The AI cites a real paper, you find the real paper, and you stop. But a real source cited for something it does not say is still a wrong answer.
Open the source and search it (Ctrl+F or Cmd+F) for the specific number, name or phrase. If the number is not there, or it is there with a different meaning (a different year, a different population, a projection rather than a result), the claim fails, however real the source is.
Worked example: an AI summary of a court order
Here is the episode from the opening, in full. On 13 September 2026 we used the AI summarising step built into our research tools to pull key facts from the sanctions order in Mata v. Avianca, a 2023 case in the Southern District of New York. We then searched the order itself for each claim. We kept the tool’s output alongside the passages it contradicts.
| The AI summary said | The order says | How we checked |
|---|---|---|
| Dated December 22, 2023 | Dated June 22, 2023 (signature block, page 34) | Searched “Dated” |
| A $5,000 sanction against “Peter Schwartz” | A $5,000 penalty imposed jointly on Peter LoDuca, Steven A. Schwartz and their law firm | Searched “$5,000” |
| Schwartz had to write to opposing counsel | Letters to the plaintiff, Roberto Mata, and to each judge falsely named as the author of a fake opinion | Read the order’s conclusion |
| The fake case was “Varghese v. Community Health Plan” | “Varghese v. China Southern Airlines Co. Ltd.” | Searched “Varghese” |
Every one of those errors read smoothly, and every one fell to step 4. A second AI summary would not have caught them. Opening the document did. Long inputs are where this is most likely: accuracy and recall degrade as a context window fills up.
Why “are you sure?” is not a check
The case behind that court order is the clearest lesson here. We told the main story in why AI makes things up: the lawyers’ brief cited six decisions that ChatGPT had invented. The detail that matters for fact-checking is what happened when doubts were raised. According to the court’s order, Steven Schwartz asked ChatGPT whether one of the cases was real and whether the others were fake. It answered that they were real and could be found on Westlaw and LexisNexis. At the sanctions hearing he testified: “I just never thought it could be made up.”
On 22 June 2023, Judge P. Kevin Castel imposed the $5,000 penalty and ordered the letters described above. Legal Dive’s coverage has the wider story.
Asking the model to confirm its own answer runs the same process that produced the answer: there is no separate step where it checks its work against anything. The same goes for AI agents, where the rule is to check the work, not the summary.
Things that feel like checking but are not
- Asking the same AI again. A regenerated answer is a new sample, not a second opinion. If a number changes between runs, neither version is evidence.
- Finding that the source exists. That is step 3. Step 4 is whether it supports the claim.
- Another AI’s summary of the source. See the worked example.
- An AI-detection tool. It guesses how text was written. It says nothing about whether a claim is true.
Make the next answer easier to check
You cannot prompt your way out of verification, but you can make it faster. Add this to requests where the facts matter:
For every factual claim in your answer, add:
- the source, with a link if you have one
- the exact sentence from that source that supports it
- or "no source" if you are relying on general knowledge
Do not invent a source to fill a gap. "No source" is an acceptable answer.Two notes on adapting it. For long answers, ask for the sources in a table at the end so the prose stays readable. And treat the quoted sentences as pointers, not proof: models can misquote as fluently as they can mis-summarise, so the quote tells you where to search, and step 4 still applies. Explicitly allowing “no source” helps for the reason covered in our prompting guide: giving the model permission to be uncertain makes the confident wrong answer less likely.
The thing worth remembering
Check the claim you would be embarrassed to repeat, in the original, before you repeat it. Most of the time it takes one search.