[ChatStream] ChatPrompt for Llama 2

Hello, this is the Product Development Department at Qualiteg.

In this article, we introduce the Llama 2-compatible ChatPrompt that is now bundled with ChatStream.

The current ChatPrompt is shown below. If you are using an older version of ChatStream, you can add Llama 2 support with the following code. (Naturally, it is already included in the latest version of ChatStream.)

from chatstream import AbstractChatPrompt
from chatstream.chat_prompt.role_type import RoleType

SYSTEM_PROMPT = """\
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\
"""


class ChatPromptMetaLlamaLlama2Chat(AbstractChatPrompt):
    """
    meta-llama/Llama-2-7b-chat

    Prompt Guide from
    https://huggingface.co/blog/llama2
    """

    def __init__(self):
        super().__init__()  # Call the initialization of the base class
        self.set_system(f"<s>[INST] <<SYS>>\n{SYSTEM_PROMPT}\n<</SYS>>\n\n")
        self.set_requester("")
        self.set_responder("")

    def get_stop_strs(self):
        if not self.chat_mode:
            return None
        return []

    def get_custom_skip_echo_len(self, skip_echo_len):
        # modify skip_echo_len when using llama2
        num_turn = self.get_turn()
        if num_turn >= 2:
            modified_skip_echo_len = skip_echo_len + 1 * self.get_turn()
            return modified_skip_echo_len
        return skip_echo_len

    def get_replacement_when_input(self):
        return None

    def get_replacement_when_output(self):  # replace when response_text gotten
        return None

    def create_prompt(self, opts={}):
        if self.chat_mode == False:
            return self.get_requester_last_msg()

        # Build the prompt for the case where Chat Mode == True
        ret = self.system

        for chat_content in self.get_contents(opts):

            chat_content_role_type = chat_content.get_role_type()
            chat_content_message = chat_content.get_message()

            if chat_content_message:
                merged_message = ""
                if chat_content_role_type == RoleType.REQUESTER:
                    merged_message = f"{chat_content_message} [/INST] "
                elif chat_content_role_type == RoleType.RESPONDER:
                    merged_message = f"{chat_content_message} </s><s>[INST] "
                ret += merged_message
            else:
                pass

        return ret

    def build_initial_prompt(self, chat_prompt):
        # No initial prompt is implemented
        pass

This implementation contains just one somewhat tricky part.
ChatStream is built around streaming chat: it delivers a real-time, flowing chat experience by updating the output one newly generated token at a time. Because of this, if the output deviates from what is expected by even a single character, it is easy to end up with a missing character during streaming, or with the entire response shifted by one character.

With Llama 2 output, we observed a phenomenon in which an extra whitespace character appears in the output with each turn. We suspected some kind of special token, an empty token, or the effects of encoding conversion, but were unable to pin down the root cause, so we added an implementation to cancel out the effect.

The phenomenon is as follows.

When the following prompt is used as the input prompt,

<s>[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>

Do you know the movie titanic [/INST] Hello! Yes, I'm familiar with the movie Titanic. It's a classic film directed by James Cameron, released in 1997, and starring Leonardo DiCaprio and Kate Winslet. The movie is based on the true story of the RMS Titanic, a British passenger liner that sank in the North Atlantic Ocean in 1912 after colliding with an iceberg. The film follows the story of Jack Dawson (played by DiCaprio) and Rose DeWitt Bukater (played by Winslet), who come from </s><s>[INST] Who is the director [/INST] The director of the movie "Titanic" is James Cameron. </s><s>[INST] Who is starred [/INST] ]  The movie "Titanic" features a star-studded cast, including:
* Leonardo DiCaprio as Jack Dawson
* Kate Winslet as Rose DeWitt Bukater
* Billy Zane as Cal Hockley
* Kathy Bates as Molly Brown
* Frances Fisher as Ruth DeWitt Bukater
* Bernard Hill as Captain Edward John Smith
* Jonathan Hyde as J. Bruce Ismay
* Eric Braeden as John Jacob Astor IV
* Gloria Stuart as Old Rose

These actors brought the characters from the movie to life and </s><s>[INST] 

the output looks like this:

[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>

Do you know the movie titanic [/INST] Hello! Yes, I'm familiar with the movie Titanic. It's a classic film directed by James Cameron, released in 1997, and starring Leonardo DiCaprio and Kate Winslet. The movie is based on the true story of the RMS Titanic, a British passenger liner that sank in the North Atlantic Ocean in 1912 after colliding with an iceberg. The film follows the story of Jack Dawson (played by DiCaprio) and Rose DeWitt Bukater (played by Winslet), who come from  [INST] Who is the director [/INST] The director of the movie "Titanic" is James Cameron.  [INST] Who is starred [/INST]

Input:
Do you know the movie titanic [/INST] Hello! Yes, I'm familiar with the movie Titanic. It's a classic film directed by James Cameron, released in 1997, and starring Leonardo DiCaprio and Kate Winslet. The movie is based on the true story of the RMS Titanic, a British passenger liner that sank in the North Atlantic Ocean in 1912 after colliding with an iceberg. The film follows the story of Jack Dawson (played by DiCaprio) and Rose DeWitt Bukater (played by Winslet), who come from [INST] Who is the director [/INST] The director of the movie "Titanic" is James Cameron. [INST] Who is starred [/INST] 

In other words, even though the input prompt contains [INST], that is, one space followed by [INST], it comes back in the output as [INST], that is, two spaces followed by [INST]. As a result, when the newly generated text is extracted, a one-character offset is observed for each response.

We then tried feeding the input prompt with two spaces followed by [INST], but in that case the generated output became three spaces followed by [INST],
so the offset was not resolved.

One option here would be to simply trim the text through ordinary text processing. However, since the policy of this ChatPrompt is to alter the original input and output as little as possible, we instead adjust skip_echo_len according to the turn, as shown below, so that the extraction lines up exactly:

    def get_custom_skip_echo_len(self, skip_echo_len):
        # modify skip_echo_len when using llama2
        num_turn = self.get_turn()
        if num_turn >= 2:
            modified_skip_echo_len = skip_echo_len + 1 * self.get_turn()
            return modified_skip_echo_len
        return skip_echo_len

Across roughly 1,000 input/output combinations, the extraction worked without issue, so we have adopted this approach as a workaround. There is a fair chance that we have simply overlooked something small, so our research team is continuing to investigate the root cause with a view to a permanent fix.

Other models that use Llama 2 as their base model exhibit the same phenomenon, and this approach resolved it for them as well.