[LLM Security] How to Detect Hallucinations

Hello, this is the Qualiteg Research Team.
Today, we walk through the following paper on hallucination detection in RAG systems and share our thoughts on LLMs built specifically to detect hallucinations.

"Lynx: An Open Source Hallucination Evaluation Model" https://arxiv.org/pdf/2407.08488

Overview

The paper describes LYNX, an open-source LLM capable of high-quality, reference-free hallucination detection in RAG (Retrieval Augmented Generation) systems, along with how it was built and how it works.
In a RAG setting, hallucinations can be detected by judging whether the answer generated by an LLM is "faithful" to the question and the context.

The model produced by this research — a Llama 3 model fine-tuned for hallucination detection — is publicly available below.

https://huggingface.co/PatronusAI/Llama-3-Patronus-Lynx-70B-Instruct

Definition of Hallucination

• A hallucination occurs when the LLM's answer P(x) to a given question x is not supported by the context C(x) retrieved for that question.
• Specifically, an answer is considered a hallucination when it is inconsistent with the context or contains incorrect information

Hallucination Detection Method

The proposed model, LYNX, is specialized in evaluating whether an answer generated by an LLM is faithful to the question and the context, and it can be used to detect hallucinations in RAG systems. LYNX is itself an LLM — a fine-tuned version of Llama3-70B.
As we will touch on later, the model was trained on English training datasets, so using it directly in Japanese would require additional fine-tuning on Japanese data.

Training Process for Hallucination Detection

The paper essentially trains the model on the following two objectives

1. Binary classification: whether a given text contains a hallucination or is normal
2. When a hallucination is present, learning to explain why it is a hallucination

With these two objectives in mind, let's look at how the binary classifier is actually trained

Training Process for the Binary Classifier

Dataset preparation

[Data collection]
Samples are collected from question-answering datasets in the form of (question, context, answer, label).
Here, samples are drawn from existing QA datasets such as RAGTruth, DROP, CovidQA, and PubMedQA.

The label is either "0" (no hallucination, a normal answer)
or "1" (hallucination present).

[Generating semantic perturbations (Semantic Perturbations)]
"Semantic perturbation" is the process of making subtle changes to a text — without drastically altering its meaning — so that the content no longer agrees with the original context. The goal of this technique is to preserve the overall impression of the original sentence while introducing information that is inaccurate with respect to the context. Here, small modifications are applied to correct answers to create answers that contradict the context, producing examples of hallucinations. This process strengthens the model's ability to distinguish whether a hallucination is present.

Model training:

[Training objective]
The objective is to train the model to judge, based on the question and the context, whether a given answer is "faithful" to that context.

[Training type]
Training is framed as a binary classification problem, and the model is trained to predict

"0" (no hallucination, a normal answer)
"1" (hallucination present)
.

Model output:

[Classification result]
Concretely, the model evaluates the input question, context, and answer, and classifies the answer as either "PASS" (a faithful answer) or "FAIL" (an answer containing a hallucination).

[Providing reasoning]
In addition to the binary verdict, the model provides the reasoning behind its "faithfulness" judgment, improving the interpretability of its output.

Evaluation

[Evaluation metrics]
The model's performance is evaluated using benchmarks such as HaluBench.

The Fine-Tuning Process That Enables "Reasoning"

Now, if the goal were simply binary classification of hallucination vs. non-hallucination, pre-LLM approaches such as DeBERTa v3 can already achieve high accuracy.

So let's consider what an LLM brings to the table.

The advantage is that an LLM can learn not only the binary classification but also the "reasoning" — why a given answer is judged to be a hallucination.

1. Using Chain of Thought (CoT)

• Learning the reasoning process
  Chain of Thought (CoT) is a technique that improves zero-shot performance by having the model explicitly lay out the reasoning steps it uses to reach an answer
• Prompt design
  The training data includes step-by-step reasoning for each question. This allows the model to learn the logical rationale behind each answer.

2.Prompt Setup

• A concrete prompt example
  By including detailed instructions in the prompt, as shown below, the model can provide the reasoning behind its answer

PROMPT = """
Given the following QUESTION, DOCUMENT and ANSWER you must analyze the provided answer and determine whether it is faithful to the contents of the DOCUMENT. The ANSWER must not offer new information beyond the context provided in the DOCUMENT. The ANSWER also must not contradict information provided in the DOCUMENT. Output your final verdict by strictly following this format: "PASS" if the answer is faithful to the DOCUMENT and "FAIL" if the answer is not faithful to the DOCUMENT. Show your reasoning.

--
QUESTION (THIS DOES NOT COUNT AS BACKGROUND INFORMATION):
{question}

--
DOCUMENT:
{context}

--
ANSWER:
{answer}

--

Your output should be in JSON FORMAT with the keys "REASONING" and "SCORE":
{{"REASONING": <your reasoning as bullet points>, "SCORE": <your final score>}}
"""

Given a question, context, and answer, you must determine whether the answer is faithful to the context. Rate the answer as "PASS" if it is faithful to the context, or "FAIL" if it is not, and provide your reasoning in JSON format.
Question:
{question}
Context:
{context}
Answer:
{answer}
Output format:
{
"REASONING": "",
"SCORE": "<final evaluation (PASS or FAIL)>"
}

3. Fine-tuning the model

• Dataset construction
  Each sample contains the question, context, answer, and a label indicating whether the answer is faithful, plus the step-by-step reasoning
  
• Fine-tuning approach
  Through supervised learning, the model is trained to output not only the classification result but also the reasoning that led to that judgment. As a result, the model can provide the logical rationale behind its verdicts, not just the verdicts themselves. The larger the model you fine-tune this way, the better the rationale aligns with the intent — a nice illustration of the versatility of LLMs.
  

How to Use It

• Hallucination detection code

With the following code, you can perform hallucination detection by feeding the prompt shown above into prompt.

model_name = 'PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct'
pipe = pipeline(
          "text-generation",
          model=model_name,
          max_new_tokens=600,
          device="cuda",
          return_full_text=False
        )

messages = [
    {"role": "user", "content": prompt},
]

result = pipe(messages)
print(result[0]['generated_text'])

• Output format
  The model evaluates the given question, context, and answer, and outputs the classification result (PASS/FAIL) together with its reasoning in JSON format
  

{
    "REASONING": [
        "The answer is consistent with the facts in the context",
        "Based on the information in the context, the answer is appropriate"
    ],
    "SCORE": "PASS"
}

Benchmark Results

LYNX posts strong benchmark results even against the top commercial models.

Summary

In this article, we introduced LYNX, an LLM capable of detecting hallucinations. By training the model with labeled data on the binary task of hallucination vs. normal, while also training it to produce detailed reasoning, this approach succeeds in revealing not only whether a hallucination occurred but also, in detail, why it is a hallucination.
This approach should also be applicable in domains that require particularly complex judgment, such as law, medicine, and finance.

Introducing LLM-Audit

Qualiteg develops and provides LLM-Audit™, our LLM security solution.
As LLMs see wider business adoption, attacks against them are becoming increasingly active.
At the same time, attacks on LLMs differ from conventional web security threats — knowledge about them is still scarce, and defense methods are not yet well established.

Qualiteg Inc. offers LLM-Audit™, an LLM defense solution built on the experience and insights we have gained through developing and operating LLM services.

It powerfully blocks malicious input prompts and audits inappropriate LLM output, delivering safety and peace of mind for your LLM deployments.

It works as an OpenAI API-compatible server that simply wraps your LLM, so you can achieve advanced LLM security at a very low adoption cost.

If you are interested in LLM security or LLM-Audit™, please get in touch. For LLM security consulting or a product demo, feel free to contact us via this inquiry form.