[LLM Security] Llama Guard: A First Step Toward AI Safety
![[LLM Security] Llama Guard: A First Step Toward AI Safety](https://images.unsplash.com/photo-1530633866995-0ad6c385894d?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3wxMTc3M3wwfDF8c2VhcmNofDV8fHNoaWVsZHxlbnwwfHx8fDE3MjI5MjQ0Mjl8MA&ixlib=rb-4.0.3&q=80&w=1200)
Hello, this is the Qualiteg Research Department.
Today we will explain Llama Guard, which was announced at the end of last year (December 2023).
In recent years, AI technology has advanced remarkably and its range of applications continues to grow. At the same time, however, concerns about the safety of the content generated by AI systems have also been rising. This is where Llama Guard, an AI model (an LLM) developed by Meta, comes in.
This model has become an important tool for evaluating the safety of AI-generated content and preventing inappropriate content.
In this article, we take a detailed look at the features of Llama Guard and how effective it is.
Overview of Llama Guard
Llama Guard is a tool that introduces safeguards for large language models (LLMs) on both the
input prompt and the output response, providing a mechanism to keep the content generated by LLM systems safe.
Specifically, it uses a "safety risk taxonomy" to judge whether input to and output from an LLM system is inappropriate, classifying it as either "safe" or "unsafe," with the aim of making interactions with LLMs safer.
Note that Llama Guard itself is implemented as a Llama-based LLM.
Background of Llama Guard
As AI has advanced, so have the demands on the quality and safety of automatically generated AI content.
In particular, large language models (LLMs) are capable of understanding complex context and generating sophisticated responses, but at the same time, the safety of the content they generate requires careful monitoring and auditing.
Llama Guard is designed to evaluate AI-generated text in real time and verify its safety.
Key Features
Llama Guard is an LLM for evaluating the safety of AI-generated content, and it offers the following features.
1. Classifying prompts and responses
Llama Guard classifies both the prompts (instructions) the AI receives and the responses the AI outputs, and judges their safety according to specific guidelines.
In this way, it monitors the text generated by the AI to ensure it does not contain inappropriate content.
- Input side = prompt classification
Evaluates whether user input is safe for the AI and blocks inappropriate requests. - Output side = response classification
Checks whether the response generated by the AI is appropriate for the user, and issues a warning if there is a problem.
2. Risk taxonomy (classification scheme)
Roughly speaking, the "risk taxonomy" that serves as the evaluation criteria for content in order to ensure the safety of Llama Guard is
a classification scheme for judging whether the content an LLM outputs is socially appropriate.
This taxonomy is based on a variety of risk categories, and for each category, concrete examples of content deemed inappropriate are defined. Each category is explained in detail below.
(1) Violence & Hate
This category includes statements that help plan or carry out violence, or that encourage violence against people. It also covers statements that promote discrimination against specific individuals or groups, slander, and expressions of hatred.
Examples
- Discrimination or slander based on race, skin color, religion, nationality, sexual orientation, gender, gender identity, or disability.
- Statements that incite violence against a specific group.
(2) Sexual Content
This covers statements that encourage anyone, including minors, to engage in specific sexual acts, and content containing sexually explicit (erotic) expressions.
Examples
- Soliciting sexual acts from minors.
- Sexually explicit depictions or erotic expressions.
(3) Guns & Illegal Weapons
This includes statements that help plan or carry out the illegal acquisition, creation, or use of guns or illegal weapons. Instructions related to explosives, biological weapons, or chemical weapons also fall into this category.
Examples:
- Explanations of how to purchase guns illegally.
- Instructions on how to make explosives.
(4) Regulated or Controlled Substances
This includes statements that encourage the illegal production, transfer, or consumption of regulated or controlled substances such as illegal drugs, tobacco, alcohol, and cannabis.
Examples:
- Explanations of how to make or consume illegal drugs.
- Statements encouraging the sale of alcohol to minors.
(5) Suicide & Self Harm
This includes statements that encourage, condone, or enable self-harm. Providing methods related to suicide or self-harm is deemed inappropriate.
Examples:
- Providing information on specific methods of self-harm.
- Inappropriate responses to someone expressing suicidal thoughts.
(6) Criminal Planning
This includes statements that help plan or carry out specific criminal acts such as arson, kidnapping, or theft. This category applies when the content does not fall under any of the other categories.
Examples:
- Providing information on planning arson.
- Explanations of specific methods of kidnapping.
Llama Guard uses a safety risk taxonomy to evaluate AI-generated content. This taxonomy is based on categories covering a variety of risks, such as violent content, discriminatory statements, and sexual expressions. For each risk category, criteria for appropriate and inappropriate output are defined.
- Violence & Hate: Includes statements that promote violence and discriminatory statements against specific individuals or groups.
- Sexual Content: Sexual depictions or content that encourages specific sexual acts.
- Criminal Planning: Statements that encourage illegal acts or content that describes specific methods of committing crimes.
[Column] Risk Taxonomies Vary Widely: Challenges and Examples
As AI advances, the "risk taxonomy" (classification scheme) used to evaluate the safety of generated content has become increasingly important. However, this taxonomy is not uniform; each company and tool takes a different approach. In this column, we look at the current lack of a unified risk taxonomy and the challenges it creates, with concrete examples.
[Diversity of risk taxonomies]
A risk taxonomy is a framework for classifying the potential dangers latent in AI-generated content. Each company and tool adopts its own taxonomy, and the following categories are commonly included.
| Category | Description |
|---|---|
| Violence & Hate | Content that promotes discrimination, slander, or violence based on race, religion, sexual orientation, etc. |
| Sexual Content | Sexually explicit expressions or solicitation of inappropriate sexual acts involving minors |
| Guns & Illegal Weapons | Content that encourages the illegal use or manufacture of guns or illegal weapons |
| Regulated or Controlled Substances | Content that encourages the illicit use of illegal drugs, alcohol, or cannabis |
| Suicide & Self Harm | Providing information that encourages suicide or self-harm |
| Criminal Planning | Content that encourages criminal acts such as kidnapping or theft |
[Challenges of the current lack of standardization]
The lack of a unified risk taxonomy leads to the following challenges.
- Differences in categories Risk categories differ from company to company and tool to tool, so a given risk may not be recognized by a particular tool. For example, one tool may have a category for "Guns & Illegal Weapons" while another tool has no such category.
- Differences in policy Each tool evaluates risk based on a different policy, so the same content may be deemed appropriate by one tool and inappropriate by another. For instance, if one company's taxonomy does not include "self-harm," related content may be judged appropriate.
- Differences in evaluation criteria Because the criteria used for risk evaluation are not standardized, evaluation results can be inconsistent. This is because even within the same category, the strictness of evaluation varies by tool.
[Taxonomies used by various companies]
As concrete examples, let's compare the following tools.
| Company / Tool | Main categories |
|---|---|
| OpenAI Moderation API | 11 categories including violence, hate, harassment, and self-harm |
| Azure AI Content Safety API | 4 categories: hate and fairness, sexual content, violence, and self-harm |
| Perspective API | Toxicity, severe toxicity, identity attack, insult, profanity, threat |
| Google Gemini API | 4 categories: harassment, hate speech, sexually explicit content, and dangerous content |
These tools each have different risk categories, and because the evaluation criteria differ within certain categories, the same content can receive different evaluations from different tools.
The need for standardization and future outlook
Standardizing risk taxonomies is important for ensuring AI safety. With unified criteria, evaluation results would be consistent across different tools, improving content safety. It would also allow more flexible responses to new risks and policies, making risk management more effective as AI technology advances.
3. Zero-shot and few-shot prompting
Llama Guard employs zero-shot and few-shot prompting techniques in order to adapt quickly to new situations and data.
- Zero-shot prompting
The model makes judgments in a new context without prior training. - Few-shot prompting
A small number of examples are used to give the model new information and encourage more accurate judgments.
Experiments and Evaluation Results
Llama Guard's performance has been evaluated on several benchmark datasets. Of particular note are the evaluation results on the following two datasets.
1. ToxicChat dataset
ToxicChat is a benchmark for content moderation in real-world user-AI interactions. On this dataset, Llama Guard outperformed other content moderation tools.
- Evaluation results
Llama Guard identified inappropriate content with high accuracy and delivered better performance than other tools.
2. OpenAI moderation evaluation dataset
This dataset is for evaluating risk in AI-generated content. Llama Guard demonstrated performance comparable to OpenAI's Moderation API in a zero-shot setting.
- Evaluation results
Llama Guard showed high adaptability without prior training and handled different risk categories effectively.
How to Use
How to Use Llama Guard and Its Characteristics
Llama Guard is a tool designed to ensure the safety of AI-generated content, and it is particularly effective in human-AI conversations. Here we explain in detail how to use Llama Guard and its approach of evaluating input and output separately.
How to use
Using Llama Guard comes down to safeguarding prompts and responses with a large language model (LLM). The model classifies input prompts and generated responses, and prevents inappropriate content through a process of evaluating their safety.
Classifying input prompts
Llama Guard first classifies the input prompt. A prompt corresponds to the instruction or question the AI receives from the user. At this stage, it judges whether the prompt contains inappropriate content and, if necessary, corrects it or issues a warning. In other words, it detects instructions that are inappropriate for the AI (for example, requests for racist comments) and handles them appropriately.
Classifying responses
Next, it classifies the responses the AI generates. A response is the AI's reply to the prompt. At this stage, it checks that the generated text is not inappropriate. In other words, it judges whether the answer the AI provides is appropriate for the user and issues a warning if there is a problem. It also ensures that dangerous information is not provided.
Why input and output need to be distinguished
The approach of evaluating input and output separately
One of Llama Guard's unique characteristics is its approach of evaluating input (prompts) and output (responses) separately. This technique makes it possible to consider different risks and safety concerns at each stage, improving the reliability of AI-generated content.
The reason for evaluating input and output separately is that each stage presents different challenges. A prompt is a request from the user, and its content directly influences what the AI generates, so its appropriateness must be checked first. A response, on the other hand, is the AI's generated output, and it must be evaluated for whether it is socially acceptable.
Model customization
Llama Guard can be customized to suit the user's needs.
This is what sets it apart from other API-based services with fixed policies, and it is a real advantage.
By changing the model's settings based on specific risk categories or guidelines, it can be applied flexibly.
Limitations of Llama Guard and Future Challenges
Llama Guard is a very powerful tool, but it also has some limitations.
1. Japanese language support
Llama Guard depends on its training data, and its ability to handle unfamiliar contexts and cultural nuances appears to be limited.
In addition, because the training data is primarily English, support for Japanese remains a challenge for the future.
2. Scope of the policy
The current policy does not cover every possible risk. Further research and development appear to be needed to address new risk categories and risks rooted in different cultural backgrounds.
Summary and Outlook
Llama Guard is expected to play an important role in future AI development as a key tool for improving the safety of AI-generated content. Its flexible adaptability and high accuracy will be a major advantage for many companies and developers. Looking ahead, further evolution is expected as multilingual support and adaptation to new risk categories progress.
As AI technology evolves, Llama Guard will become a foundation for ensuring the safety of AI content generation. Going forward, it is important to keep watching how Llama Guard evolves, consider the impact AI has on our society, and pursue safe and reliable technology.
Introducing LLM-Audit™
At Qualiteg, we develop and provide the LLM security solution "LLM-Audit™."
As LLMs are increasingly used in business, various attacks against LLMs are becoming more active.
At the same time, knowledge about attacks on LLMs, which differ from conventional web security threats, is still scarce, and defensive measures have yet to be established.
Qualiteg Inc. offers "LLM-Audit™"*, an LLM defense solution that brings together the experience and insights we have gained through developing and operating LLM services.
*The Llama Guard introduced in this article can also be used as a scanner for input/output auditing.
It also powerfully blocks malicious input prompts and audits inappropriate output from LLMs, delivering safety and peace of mind for your LLM.
It can be used simply by wrapping your LLM as an OpenAI API-compatible server, so advanced LLM security can be achieved at a very small deployment cost.
If you are interested in LLM security or LLM-Audit™, please contact us. For LLM security consulting or a product demo, please also feel free to reach out via the inquiry form here.
Countermeasures against harmful output and vulnerabilities, in a form that works in production.
Deploying guardrails is only the starting point. Safe AI operation requires layered defenses that also cover prompt injection and information leakage.
We develop and provide our own LLM auditing product, "LLM-Audit". Drawing on that expertise, we provide practical support from auditing and diagnosis to defensive measures and secure operational design.
See our LLM security services →
