Why Cited Answers Still Go Wrong

Citations make AI search answers look trustworthy. Each claim has a little link. It feels like someone did the homework.

But citations can be wrong in ways that are hard to see. This lesson names six common failures, explains where each one comes from in the pipeline, and shows what each looks like in a real answer.

What You'll Learn

• Six common ways a cited AI search answer goes wrong
• Which step of the pipeline causes each one
• What each failure looks like when you read an answer
• Why a missing answer is safer than a confident wrong one

Failure 1: The citation points to a page that does not say it

This is one of the most common problems. The answer makes a claim, the citation links to a real page, but the page does not contain that claim. In a 2023 Stanford study that checked AI search answers by hand (Liu, Zhang and Liang, "Evaluating Verifiability in Generative Search Engines"), only about three in four citations in those 2023-era tools supported the sentence they were attached to. That is often enough that you should expect it.

Where it comes from: the write-and-cite step. As you saw in Lesson 2, a citation means a passage was used near that sentence. The model may have added a detail from memory (Lesson 3), stretched what the passage said, or attached the citation to the wrong sentence.

What it looks like:

"The museum is free on the first Sunday of every month [2]."

You open source 2. It is the museum's page. It lists opening hours and ticket prices. It says nothing about free Sundays.

The claim might be true. It might be old. It might be from a different museum. You cannot tell from the citation, because the citation does not support it.

Failure 2: Stale sources

Where it comes from: the retrieve step. Search ranking likes pages that many sites link to, and old pages have had more time to collect links. So an old article can outrank the current one. The passage may also have no date near it, so the model cannot tell it is old.

What it looks like: an answer gives a price, a rule, a deadline, or a product feature that was true two years ago. The cited page is real and does say it. But the page is out of date.

Stale sources are a big problem for anything that changes: prices, laws, visa rules, software features, app settings, sports results, and "the latest" anything.

Failure 3: Low-quality sources

Where it comes from: the retrieve and rank step. The web is full of pages written to rank in search, not to be correct. Some are copied from other pages. Some are written by AI with no checking. Some are forum posts with one person's guess.

What it looks like: the citation goes to an unknown blog, a content farm, or a forum thread. The answer repeats its claim in the same calm tone it would use for a government page or a research paper. The writing style of the answer does not change with the quality of the source.

Failure 4: Merging two sources into a false claim

Where it comes from: the pick-passages and write steps together. The model reads short passages from different pages side by side. It tries to write one smooth answer. Sometimes it joins two true facts into one false one.

What it looks like:

• Source 1 says: "Company A raised prices by 10% in March."
• Source 2 says: "Company B's new plan costs 15 dollars a month."
• The answer says: "Company A raised its plan to 15 dollars a month in March [1][2]."

Both citations are real. Both pages are correct. The sentence is wrong. This failure is hard to catch, because each source checks out on its own. You only find it when you look for the exact claim on each page.

Other forms of the same problem: mixing up two people with the same name, two products with similar names, or numbers from two different years.

Failure 5: A confident summary of a weak source

Where it comes from: the write step. The model's job is to write a clear, direct answer. A source might say "some early reports suggest this may help." The summary turns it into "this helps."

What it looks like: a strong, simple claim backed by one source. When you open the source, it is careful, uncertain, or talks about a small test. The hedges and limits did not make it into the summary.

This also happens when only one page on the web discusses a topic. The answer looks the same as one built from ten agreeing sources.

Failure 6: Answering when no good source exists

Where it comes from: the whole pipeline. Every step is built to produce an answer. If the search finds nothing useful, the tool still has some passages, and the model still writes.

What it looks like:

• A question about a local, brand new, or niche topic gets a full answer.
• The citations are only loosely related to the question.
• The answer is vague, or it answers a slightly different question than the one you asked.

A good tool says "I could not find a reliable source for this." Not every tool does, and not every time. When the honest answer is "no one has published this," a confident answer is a warning sign.

Where each failure comes from

Try it: break an answer into claims

The first step in checking any answer is to see each claim on its own. A long paragraph hides them. Paste an AI search answer below the prompt and run it.

Now you have a checklist. Every claim with NONE is likely from the model's memory. Every claim with a citation is something you can look for on that page. The next lesson shows how to do that quickly.

Key Takeaways

• A citation can link to a real page that does not contain the claim. This is one of the most common failures.
• Stale and low-quality pages can rank well, and the answer's tone does not change with source quality.
• The model can merge two true facts from two sources into one false claim, and both citations will look fine.
• Summaries often drop hedges, turning "may help" into "helps."
• When no good source exists, many tools still answer. A confident answer on a niche topic is a warning sign.
• Splitting an answer into separate claims is the first step to checking it.