From Newsgroup: uk.media.radio.archers
On 14/12/2025 17:04, BrritSki wrote:
Just for a laugh I played with ChatGPT today, starting with something
I'm quite an expert on:
<Snip of AI generated nonsense.>
I read somewhere that AI is designed to give an answer and make you
happy rather than to tell the truth. To get good results you need to
give it some context and an expectation of what you want and tell it to
be honest if it doesn't know.
Then a had an idea: I asked Google Gemini. It said:
Getting the most helpful and truthful answers from an AI isn't just
about asking a question; it's about how you frame it. AI models are
pattern matchersrCothey predict the next most likely word based on your
input. To get the best results, you need to guide that prediction process.
Here are the most effective strategies to maximize accuracy and helpfulness:
1. "Chain of Thought" Prompting
This is arguably the single most effective technique for complex
reasoning. Instead of asking for just the answer, ask the AI to show its
work.
* How to do it: Add phrases like "Think step-by-step," "Explain your
reasoning before giving the final answer," or "Break this down into its component parts."
* Why it works: When an AI "thinks" out loud, it generates its own
context. This intermediate text helps the model stay on track and often
allows it to self-correct errors before arriving at the final conclusion.
2. Assign a Persona or Role
giving the AI a specific identity helps narrow down the tone,
complexity, and domain of knowledge it draws from.
* How to do it: "Act as a senior Python developer," "You are an
empathetic grief counselor," or "Explain this as if you are a physics professor teaching a 101 class."
* Why it works: It sets the expectations for the output. A "lawyer"
persona will be more cautious and precise; a "creative writer" persona
will be more flowery and inventive.
3. Explicit Constraints on Truthfulness
AI models can "hallucinate" (confidently state false information). You
can reduce this by giving explicit instructions on how to handle
uncertainty.
* How to do it: "If you do not know the answer, say 'I don't know'
rather than guessing." "Only use the provided text to answer the
question." "Cite your sources if possible."
* Why it works: It overrides the model's default tendency to always
provide a completion, prioritizing accuracy over fluency.
4. Few-Shot Prompting (Providing Examples)
Showing the AI exactly what you want is often better than telling it.
* How to do it
* *User:* "Convert these movie titles into emojis."
* *User:* "Titanic -> EfUorYaN+AEfAo"
* *User:* "The Lion King -> EfaUEfaaEfia"
* *User:* "Star Wars ->"
* Why it works: It establishes a clear pattern for the AI to follow, significantly reducing the chance of misinterpretation.
5. Give Context and Background
Ambiguity is the enemy of accuracy. The more context you provide, the
less the AI has to guess.
* Instead of: "Write an email to my boss."
* Try: "Write an email to my boss, Sarah. I need to ask for next Friday
off because I have a dentist appointment. Keep it professional but brief."
6. Iterative Refinement
Don't expect the first answer to be perfect. Treat the AI as a collaborator.
* How to do it: If the answer is vague, say: "That's too general. Can
you be more specific about [X]?" If it's too complex, say: "Simplify
this for a non-expert."
* Why it works: You can steer the AI back on course if it drifts or misunderstood your initial intent.
Summary: The "Mega-Prompt" Structure
For critical or complex questions, you can combine these techniques into
one structured prompt:
1. Role: "Act as an expert in..."
2. Context: "I am trying to..."
3. Task: "Please provide a..."
4. Constraints: "Do not guess. If you are unsure, state that."
5. Format: "Format the answer as a bulleted list."
6. Reasoning: "Walk me through your logic step-by-step."
By reducing ambiguity and encouraging the AI to process information
logically, you significantly increase the odds of getting a
high-quality, truthful response.
Then I asked how I could be sure of the respose, and it said:
The honest answer is: You cannot be 100% sure based on the AIrCOs response alone.
As we saw with the Agatha Christie tables earlier, AI can be incredibly confident, perfectly formatted, and completely wrong. Large Language
Models (LLMs) are designed to sound plausible, not necessarily to be
factual. They are predicting the next word, not accessing a database of absolute truth.
Here is a guide on how to verify AI responses and spot when it might be "hallucinating" (making things up).
1. The "Trust but Verify" Protocol
If the information is important (medical, legal, financial, or specific historical data), you must verify it externally.
* Ask for Sources: Ask the AI, "Please list the specific sources or URLs
you used for this."
* *Warning:* AI can sometimes fabricate sources (e.g., inventing a
court case or a news article). Always click the link or search for the
title to ensure it exists.
* Cross-Reference: Take the specific names, dates, or facts the AI gave
you and plug them into a search engine (Google, Bing, etc.). If the AI mentions a specific event (like "The 1849 attempt on Queen Victoria"), a
quick search should confirm it immediately.
2. The "DevilrCOs Advocate" Prompt
AI models are eager to please. If you ask, "Is X true?" they often lean towards "Yes." To test the truth, ask the AI to argue against itself.
* The Prompt: "Are there any alternative views on this?" or "Critique
your previous answer. Are there any inaccuracies or missing context?"
* Why it works: This forces the model to re-evaluate its own output. If
I had done this with the Agatha Christie table, I might have caught that
I labeled everyone a suicide.
3. Check for "Hallucination" Signs
There are subtle "tells" that an AI is making things up:
* Vague Authority: Phrases like "Experts suggest..." or "Studies
show..." without naming the experts or the studies.
* Repetitive Phrasing: If the AI repeats the same point three times in slightly different words, it often means it lacks actual depth on the topic.
* Logical Inconsistencies: Like the Christie list where historical
deaths were *all* suicides. If a pattern looks too perfect or rigid, it
might be a formatting error rather than factual reality.
4. Know the "Danger Zones"
AI is generally reliable for some things and unreliable for others.
* High Reliability: Summarizing text you provide, coding (mostly),
explaining established scientific concepts (e.g., "How does gravity
work?"), creative writing.
* Low Reliability: Obscure quotes (it often misattributes them),
specific numbers/dates in niche history, recent events (unless it has a
search tool), and math (it often struggles with complex calculation
unless using a calculator tool).
5. Use the "Grounding" Technique
If you want the truth based on a specific document, paste the text into
the chat and give this instruction:
* The Prompt: "Answer the following question using only the text
provided above. If the answer is not in the text, state that you do not
know."
* Why it works: This prevents the AI from bringing in outside
"hallucinations" and forces it to act as a strict analyst of the data
you gave it.
Summary
Treat the AI like a highly intelligent, well-read, but occasionally
exhausted research assistant. It is brilliant at gathering and
formatting information, but you are the editor. You must sign off on the
final accuracy.
Now should I believe all that? (The references to Agatha Christie relate
to a previous question I had asked where it made a total hash of the response.)
--
Ric
--- Synchronet 3.21a-Linux NewsLink 1.2