[LLM Security] Zero-Resource Black-Box Hallucination Detection

Hello from the Qualiteg Research Team.

In this article, we walk through the following paper, which proposes a method for detecting hallucinations in a "zero-resource" setting, i.e., without relying on external information such as a database.

SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
https://arxiv.org/abs/2303.08896

Background: The Problem of LLM "Hallucinations"

In recent years, generative large language models (LLMs) have become capable of producing highly fluent responses to a wide variety of user prompts.
However, these models are known to hallucinate facts and make non-factual statements, which can undermine trust in their output.
Existing fact-checking methods for addressing this problem either require access to the output probability distribution or rely on complex modules that use external databases.

This paper proposes a new approach called SelfCheckGPT. It is a simple sampling-based method for fact-checking the responses of black-box models with zero resources.
SelfCheckGPT is based on a simple idea: if an LLM has knowledge of a given concept, its sampled responses are likely to be similar and to contain consistent facts. For hallucinated facts, on the other hand, stochastically sampled responses tend to diverge and may contradict one another.

Roughly speaking, here is how the hallucination detection works.

First, a brief explanation of the principle.

1. Have the LLM produce a response to a given input prompt.
2. Then, using the same input prompt, have the LLM produce several more responses.
3. Compare the multiple output texts with the original output. If they all say the same thing, there is "no hallucination"; if the contents vary when the outputs are compared with one another, there is a "hallucination".
4. Evaluating whether the outputs vary or not is called "consistency evaluation". The paper explains that there are several methods for evaluating consistency, each with its own strengths and weaknesses.

Let's now look at this in a little more detail.

Advantages of Hallucination Detection with This Method

The advantage is that hallucinations can be detected without any elaborate machinery, as described below.

• No external database required
  Because the model's knowledge can be evaluated by sampling alone, the method can be applied to black-box models without using external resources.
• Zero-resource approach:
  It does not depend on the model's internal probability distribution or on external data; it evaluates purely on the agreement and consistency of the generated text.

Generating Diverse Responses by Sampling

First, multiple responses are generated from the LLM (large language model) for a given user prompt. Diversity is introduced by sampling the responses stochastically.

(Here, "sampling" refers to generating multiple responses using the same prompt.)

Sampling procedure

• A specific prompt is given to the language model, and the responses the model generates are obtained multiple times. This produces variation among the responses.
• For example, given the prompt "What is John Smith's occupation?", the model may generate several different occupations.

Using the temperature parameter

• During sampling, the diversity of responses is controlled by adjusting the temperature parameter. A higher temperature increases randomness and yields more diverse responses.
• In the paper, the main response is generated with standard beam search at temperature 0.0, while the sample responses are generated at temperature 1.0 to obtain diverse responses.

Consistency Evaluation

The multiple generated sample responses are compared to measure how consistent they are with one another. Information that is consistent is likely to be factual, while information that varies may be a hallucination.

Methods used for consistency evaluation include BERTScore, question answering, n-gram models, natural language inference (NLI), and prompt-based evaluation. For example, with BERTScore, the similarity between sentences in the response and sentences in the sample responses is measured using BERT to evaluate consistency. With question answering, questions are automatically generated from the main response, and the method checks whether the answers to those questions agree in the sample responses as well. With n-gram models, a model built from the samples is used to evaluate the probability of occurrence of the sentences in the original response; a low probability is taken to indicate a likely hallucination. With NLI, the method evaluates whether the response contradicts the samples, and if there are many contradictions it is treated as a hallucination. Finally, with prompt-based evaluation, an LLM is asked to judge with Yes/No whether a sentence is supported by the samples, and consistency is measured accordingly.

Summary of Consistency Evaluation Methods

The table below summarizes the evaluation methods for checking consistency, i.e., whether the content varies.

Computing the Hallucination Score

Based on the consistency information obtained in this way, a hallucination score is computed for each sentence. The score ranges from 0.0 to 1.0: the closer to 0.0, the more factual; the closer to 1.0, the more likely it is a hallucination. The basic principle behind the detection is that when the LLM has accurate knowledge about a particular piece of information, the sampled responses tend to be similar and to contain consistent facts. Conversely, hallucinated facts vary across the sample responses and are prone to contradiction. In this way, SelfCheckGPT takes an approach that detects hallucinations based on the consistency of responses.

Conclusion and Recommended Methods

• The most effective method
  According to the paper's results, the prompt-based method (SelfCheckGPT with Prompt) shows the highest accuracy and is the best choice especially when using newer language models (for example, GPT-3.5 or later). Because it delivers highly consistent evaluations, it is reported to be the most reliable for hallucination detection.
  
• Combining multiple methods
  Since each method offers a different approach, combining several methods can further improve detection accuracy. In particular, the combination of the prompt-based method and the NLI method was found to strike a good balance between performance and computational cost and to be applicable to a wide range of scenarios.

Summary

Since computational cost and available resources differ depending on the actual usage scenario, which method to choose depends on the situation. Ultimately, though, the following points are worth considering when making that choice.

1. When high accuracy is required:
   Use the prompt-based method as the mainstay, supplemented with the NLI method as needed.
2. When you want to keep computational cost down
   Use a combination of the n-gram method and the BERTScore method.
3. When fine-grained verification is needed:
   Use the question-answering method to perform detailed consistency evaluation.

Introducing LLM-Audit™

Qualiteg develops and provides LLM-Audit™, our LLM security solution.
As LLMs see wider business adoption, attacks against them are becoming increasingly active.
At the same time, attacks on LLMs differ from conventional web security threats — knowledge about them is still scarce, and defense methods are not yet well established.

Qualiteg Inc. offers LLM-Audit™*, an LLM defense solution built on the experience and insights we have gained through developing and operating LLM services.

* It also supports hallucination detection with SelfCheckGPT as described in this paper.

It powerfully blocks malicious input prompts and audits inappropriate LLM output, delivering safety and peace of mind for your LLM deployments.

It works as an OpenAI API-compatible server that simply wraps your LLM, so you can achieve advanced LLM security at a very low adoption cost.

If you are interested in LLM security or LLM-Audit™, please get in touch. For LLM security consulting or a product demo, feel free to contact us via this inquiry form.