A few days ago Andrej Karpathy tweeted:
On the “hallucination problem”
I always struggle a bit with I’m asked about the “hallucination problem” in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines.
We direct their dreams with prompts. The prompts start the dream, and based on the LLM’s hazy recollection of its training documents, most of the time the result goes someplace useful.
It’s only when the dreams go into deemed factually incorrect territory that we label it a “hallucination”. It looks like a bug, but it’s just the LLM doing what it always does.
At the other end of the extreme consider a search engine. It takes the prompt and just returns one of the most similar “training documents” it has in its database, verbatim. You could say that this search engine has a “creativity problem” – it will never respond with something new. An LLM is 100% dreaming and has the hallucination problem. A search engine is 0% dreaming and has the creativity problem.
All that said, I realize that what people actually mean is they don’t want an LLM Assistant (a product like ChatGPT etc.) to hallucinate. An LLM Assistant is a lot more complex system than just the LLM itself, even if one is at the heart of it. There are many ways to mitigate hallcuinations in these systems – using Retrieval Augmented Generation (RAG) to more strongly anchor the dreams in real data through in-context learning is maybe the most common one. Disagreements between multiple samples, reflection, verification chains. Decoding uncertainty from activations. Tool use. All an active and very interesting areas of research.
TLDR I know I’m being super pedantic but the LLM has no “hallucination problem”. Hallucination is not a bug, it is LLM’s greatest feature. The LLM Assistant has a hallucination problem, and we should fix it.
Okay I feel much better now 🙂
Andrej Karpathy @karpathy
I truly appreciate your recognition of the differences between how large language models (LLMs) work and how traditional search engines function. It’s fascinating how LLMs have revolutionized various fields, including the creative realm. In creative endeavors, like writing poems, short stories, or even crafting an imaginative piece of fiction, the so-called “hallucination problem” of LLMs can prove to be surprisingly advantageous.
When you engage in creative writing, your primary objective is not to adhere strictly to accuracy and factual representation but rather to explore the limitless boundaries of your imagination. LLMs, with their ability to generate creative and unexpected content, can be a valuable tool to tap into new ideas and inspire innovative storytelling. They can help writers break free from conventional thinking patterns and venture into unexplored territories, allowing their creativity to flourish.
Conversely, in more formal and specialized writing contexts, such as drafting legal briefs or preparing technical reports, accuracy and precision are of paramount importance. LLM hallucinations, where the models generate content that may not be factually correct or contextually appropriate, cannot be tolerated in such situations. Here, the purpose is to convey information accurately, adhere to specific guidelines, and present a strong and well-supported argument.
It’s intriguing how the same technology that opens doors to unprecedented creative possibilities can also present challenges in other domains where accuracy and reliability are crucial. This duality highlights the importance of understanding the appropriate use cases for LLMs and being cognizant of the potential pitfalls and limitations they may possess in certain instances.
In summary, the LLM hallucination problem can indeed prove beneficial when the goal is creative expression, enabling writers to push boundaries and explore unconventional ideas. However, in situations that demand accuracy and precision, such as legal or technical writing, it becomes imperative to approach LLM-generated output with caution and verification to ensure the information presented is reliable and contextually appropriate.