Hello from the Product Development Team at Qualiteg.

In this article, we introduce a technique you can use when loading an LLM takes a long time.

For testing, or when implementing a sample app that uses the ChatStream API, there are situations where you want to restart ChatStream several times and try things out. In such cases, you often do not really need to load an actual LLM; a "canned" response that follows the ChatStream API is perfectly fine.

One trick here is to load a small LLM to shorten the loading time, but if all you need is a fixed response, you can simply specify a Mock response instead.

If you have found the time spent loading the LLM on every restart stressful, please give this a try.

Using Generator Mock Responses (Fast Startup)

With Generator Mock responses, you can have dummy text generated in place of a pre-trained language model that takes a long time to load.

Usage

chat_stream = ChatStream(
    use_mock_response=True,
    mock_params={"type": "echo", "initial_wait_sec": 1, "time_per_token_sec": 1},
    chat_prompt_clazz=ChatPrompt,
)

Constructor arguments of the ChatStream class

• use_mock_response ... True enables generator mock responses.
• mock_params ... Specifies the rules for generating generator mock responses.
• chat_prompt_clazz ... The prompt history management class.

mock_params parameters

With settings like the above, ChatStream does not need to load an LLM, so it starts up in an instant.

Difference from Transformer Mock

A similar feature is Transformer Mock mode.

https://journal.qualiteg.com/transformerying-da-womotukusuru/

Whereas Transformer Mock mode records and replays the behavior of the actual Model and Tokenizer, Generator Mock
responds with dummy text after receiving input. Transformer Mock mode only accepts the predetermined inputs, while Generator Mock
responds with dummy text to any input whatsoever.
You can use whichever suits your purpose.