What Is ChatPrompt?

Hello, this is the Product Development Department at Qualiteg Inc.

In this article, we walk through how to implement a ChatPrompt in concrete terms.

ChatPrompt is a class for generating prompts for a pretrained language model (hereafter, "the model"). We call it a prompt class.

For example, in the case of redpajama-incite, you build a prompt like the following and feed it to the model.

<human>: Who is Alan Turing
<bot>:

The model then generates the continuation and outputs the following.

<human>: Who is Alan Turing
<bot>: He was a very honorable man.

In this example, <human> and <bot> are each followed by :, and turns are separated by \n.

These rules and conventions differ subtly from model to model.

The class that generates such prompts and maintains the conversation history is ChatPrompt, and we refer to it as a prompt class.

As mentioned above, since each model has its own conventions, a separate prompt class is required for each model.

Preset Prompt Classes

ChatStream ships with prompt classes (ChatPrompt classes) for several well-known models.

Writing Your Own Prompt Class

If no prompt class exists yet, for example for a new model, you can write your own.

Implementing a ChatPrompt Prompt Class

The responsibilities of a prompt class are as follows:

• 1. Output a prompt for the model based on the user's input and the conversation history so far
• 2. To generate text properly,
  • 2-1. Hold information about the special tokens that stop text generation
  • 2-2. Hold conversion information for specific tokens

Because prompt conventions differ by model, you implement those differences here.

That said, it is not difficult: essentially, you are just defining the rules for concatenating text.

Importing the Base Class for Prompt Classes

Import the AbstractChatPrompt class, which serves as the base for prompt classes.

from chatstream import AbstractChatPrompt

Overriding the Base Class

AbstractChatPrompt is an abstract class, so you override the required methods.

Below is an example implementation of a prompt class for rinna/japanese-gpt-neox-3.6b-instruction-sft.

For this model, the goal is to output a prompt in the following format (the role names ユーザー = "User" and システム = "System" are part of the model's specification, so they are kept in Japanese):

ユーザー: 日本のおすすめの観光地を教えてください。<NL>システム: どの地域の観光地が知りたいですか？<NL>ユーザー: 渋谷の観光地を教えてください。<NL>システム: 

(Meaning: User: Please tell me some recommended sightseeing spots in Japan.System: Which region are you interested in?User: Please tell me about sightseeing spots in Shibuya.System: )

The implementation looks like this:

from chatstream import AbstractChatPrompt


class ChatPromptRinnaJapaneseGPTNeoxInst(AbstractChatPrompt):
    def __init__(self):
        super().__init__()
        self.set_requester("ユーザー")  # Role name of the side making requests to the model ("User" in Japanese; required by this model)
        self.set_responder("システム")  # Role name of the responding side, i.e., the model ("System" in Japanese; required by this model)

    def get_stop_strs(self):
        return []  # If you want to stop generation when a certain keyword appears, list those keywords here

    def get_replacement_when_input(self):
        return [("\n", "<NL>")]  # Replacement rules for input text at input time

    def get_replacement_when_output(self):
        return [("<NL>", "\n")]  # Replacement rules for output text at output time

    def create_prompt(self, opts={}):
        # Build the prompt
        ret = self.system;
        # Get the list of conversation history so far with get_contents
        for chat_content in self.get_contents(opts):
            # Get the role name
            chat_content_role = chat_content.get_role()
            # Get the message
            chat_content_message = chat_content.get_message()

            if chat_content_role:

                if chat_content_message:
                    # If a message part exists
                    merged_message = chat_content_role + ": " + chat_content_message + "<NL>"
                else:
                    merged_message = chat_content_role + ": "

                ret += merged_message

        return ret

    def build_initial_prompt(self, chat_prompt):
        # No initial prompt is implemented
        pass

Implementing the Prompt Class: Setting the Roles

• In the constructor, call the base class's __init__().
• For a two-party chat, specify the role names with set_requester and set_responder.

def __init__(self):
    super().__init__()
    self.set_requester("ユーザー")  # Role name of the side making requests to the model ("User" in Japanese; required by this model)
    self.set_responder("システム")  # Role name of the responding side, i.e., the model ("System" in Japanese; required by this model)

If you need a system-wide initialization message, set the system message with the set_system method.

def __init__(self):
    super().__init__()
    self.set_system("ユーザーとシステムからなるチャットシステムです。システムはユーザーに対して丁寧かつ正確な回答をするよう心がけます")  # Japanese system message: "This is a chat system consisting of a user and a system. The system strives to give the user polite and accurate answers."
    self.set_requester("ユーザー")  # Role name of the side making requests to the model ("User" in Japanese; required by this model)
    self.set_responder("システム")  # Role name of the responding side, i.e., the model ("System" in Japanese; required by this model)

Implementing the Prompt Class: Setting Stop Strings

• By specifying stop strings, you can stop text generation when a specific keyword or token appears.
• If none are needed, use return [].
• Stop strings are different from the EOS token. Even if you specify no stop strings here,
  text generation will still stop at the EOS token preconfigured in the tokenizer (tokenizer.eos_token_id).

    def get_stop_strs(self):


    return ['</s>']  # Stop text generation when '</s>' appears

Implementing the Prompt Class: Setting Replacement Rules for Input Text

In a chat implementation, the text entered by the user is basically fed to the model as-is, but there may be characters the model cannot accept, or characters that need to be converted before being fed to the model.

For example, if the user's input contains \n (a newline) but the model cannot accept \n, you need to replace \n with an appropriate string.

To perform replacements when feeding user input to the model, specify them as follows:

    def get_replacement_when_input(self):


    return [("\n", "<NL>")]  # Replacement rules for input text at input time

Here we specify that \n should be replaced with <NL>. Each pair is specified as a tuple, such as ("\n", "<NL>").
If you want to register multiple replacement patterns, specify multiple tuples.

Implementing the Prompt Class: Setting Replacement Rules for Output Text

You can also replace keywords that appear in the text output by the model.

For example, if the model's output is おはようございます。<NL>何か御用でしょうか ("Good morning.How may I help you?") and you want to replace <NL> with \n to represent a newline, you can configure the output replacement as follows:

   def get_replacement_when_output(self):


   return [("<NL>", "\n")]  # Replacement rules for output text at output time

Implementing the Prompt Class: Generating the Prompt

The create_prompt method generates the entire prompt, including the past conversation history.

The past conversation history can be obtained with self.get_contents().

The return value of get_contents is a list whose elements are instances of the ChatContent class.

A ChatContent instance holds a single chat entry; chat_content.getRole() returns the role name, and chat_content.get_message()
returns what that role said (the text).

By writing the logic that concatenates these, you can generate a prompt in the format the model expects.

    def create_prompt(self, opts={}):


# Build the prompt
ret = self.system;
# Get the list of conversation history so far with get_contents
for chat_content in self.get_contents(opts):
    # Get the role name
    chat_content_role = chat_content.get_role()
    # Get the message
    chat_content_message = chat_content.get_message()

    if chat_content_role:

        if chat_content_message:
            # If a message part exists
            merged_message = chat_content_role + ": " + chat_content_message + "<NL>"
        else:
            merged_message = chat_content_role + ": "

        ret += merged_message

return ret

Implementing the Prompt Class: Generating the Initial Prompt and Initial Context

Depending on the model, you may want to set up some conversational context in advance.

Feeding input to the model and having it generate text right away is called zero-shot,
but providing some input beforehand, such as background knowledge or examples, can sometimes make subsequent output more stable.

This is referred to as one-shot or few-shot.

In a chat setting, it is also used for purposes such as starting the conversation from a particular topic.

The following is an example of using the initial context to start the chat as if the conversation were already about the movie "Titanic."

Override the build_initial_prompt method.

def build_initial_prompt(self, chat_prompt):
    chat_prompt.add_requester_msg("Do you know about the Titanic movie?")
    chat_prompt.add_responder_msg("Yes, I am familiar with it.")
    chat_prompt.add_requester_msg("Who starred in the movie?")
    chat_prompt.add_responder_msg("Leonardo DiCaprio and Kate Winslet.")