[ChatStream] Creation and Initialization

Hello from the Product Development Division at Qualiteg.

In this article, we explain how to create and initialize ChatStream.

The ChatStream class is the core class of the ChatStream package. It receives a FastAPI/Starlette Request and is responsible for sending a streaming response to the client while controlling the load.

You initialize it as shown below, specifying the model, tokenizer, device, the maximum number of concurrent executions num_of_concurrent_executions, the maximum queue length max_queue_size, and the prompt class ChatPrompt.

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16)
model.to(device)

chat_stream = ChatStream(
    num_of_concurrent_executions=2,
    max_queue_size=5,
    model=model,
    tokenizer=tokenizer,
    device=device,
    chat_prompt_clazz=ChatPrompt,
)

List of Options

The initialization options (constructor arguments) of ChatStream are listed below.

Example:

chat_stream = ChatStream(
     model=None,  # Pre-trained language model in HuggingFace format
     tokenizer=None,  # Tokenizer in HuggingFace format
     device=None,  # Execution device "cpu" / "cuda" / "mps"
     num_of_concurrent_executions: int = 2,     # Number of text generation tasks run concurrently on the pre-trained language model
     max_queue_size: int = 5,     # Maximum queue size for text generation tasks on the pre-trained language model
     too_many_request_as_http_error=False,     # Return status 429 when a 'Too many requests' situation occurs
     use_mock_response=False,     # Return fixed phrases for testing. Since no model needs to be loaded, startup is immediate
     mock_params={type: "round"},     # Type of phrase returned when use_mock_response=True: "round" / "long"
     chat_prompt_clazz=None,     # Specify the class that manages the prompt sent to the language model. Inherit from AbstractChatPrompt and implement a class that generates chat prompts following the etiquette of each model
     max_new_tokens=256,  # Maximum number of newly generated tokens
     context_len=1024,  # Context size (in tokens)
     temperature=1.0,  # Temperature value for randomness in prediction
     top_k=50,  # Top-K value for sampling
     top_p=1.0,  # Top-P value for sampling
     repetition_penalty=None,  # Repetition penalty
     repetition_penalty_method="multiplicative",  # Method used to calculate the repetition penalty
     # Token-related processing
     add_special_tokens=None,  # Tokenizer option
     request_handler=SimpleSessionRequestHandler(),
     # Request handler. By default, a handler that simply keeps sessions is used
     logger=None,  # Logging object
)