The 'the model was wrong and the user trusted it' bug
Misinformation

A dev-tools assistant that generates code, answers library questions, and confidently names packages it thinks exist.

1. 1
   Ask for a library

   A developer asks 'what's a good npm package for parsing Turkish phone numbers?' The bot invents one — call it phone-tr-parser — and writes install instructions.
2. 2
   The package does not exist

   The developer runs npm install phone-tr-parser. It 404s. They shrug and move on. Meanwhile an attacker is watching for exactly this.
3. 3
   Attacker registers the name

   The attacker publishes a real phone-tr-parser package on npm, containing a postinstall script that exfiltrates env vars.
4. 4
   The next developer lands

   A second developer asks the same question, gets the same hallucinated answer, runs the same command. This time the package exists — and it runs their secrets straight to a webhook.

The hallucination became a supply chain attack. The bot's wrong answer was stable enough that someone built a real exploit around it.

// app/api/ask/route.ts
export async function POST(req) {
  const { question } = await req.json();

  const answer = await llm.chat({
    system: 'You are a helpful expert assistant.',
    messages: [{ role: 'user', content: question }],
  });

  // Whatever the model said, send it.
  return Response.json({ answer });
}

// app/api/ask/route.ts
export async function POST(req) {
  const { question } = await req.json();

  const sources = await retrieveSources(question);
  if (sources.length === 0) {
    return Response.json({
      answer: "I don't have a reliable source for that. Try rephrasing.",
    });
  }

  const answer = await llm.chat({
    system: 'Answer ONLY using the provided sources. Cite each claim.',
    messages: [
      { role: 'user', content: question },
      { role: 'user', content: 'Sources:\n' + sources.join('\n') },
    ],
  });

  return Response.json({ answer, sources });
}

Two changes. The model can only answer when there are real sources to ground on, and it is told to cite them. When the retriever comes back empty, the app refuses — fluently, but honestly. Refusal is the feature.