Qualiteg - ChatStream Guide - Qualiteg Journal

LLM

LLM Inference Infrastructure Provisioning Course, Part 4: Selecting an Inference Engine

Hello! In the previous installments of this course, we covered in detail how to estimate the request volume an LLM service needs to handle and how to calculate the memory a model consumes during inference. This time we dig into the fourth of the seven steps: selecting an inference engine.

ChatStream Guide

chatstream.net Query Parameter Specification

chatstream.net is a service operated by Qualiteg Inc. that lets you try out the latest publicly available LLMs from around the world as soon as they are released. You can open a particular LLM for an in-depth chat, open several LLMs and use them together, or have LLMs compare

ChatStream Guide

Three Penalties in LLM Sampling

[Appendix] Comparison of Penalties Penalty type Purpose How it is applied Example penalty Repetition Penalty Prevents specific tokens or phrases from being repeated. Applies a penalty to the log probabilities (logits) of all previously generated tokens. For example, if a token has already been generated, its log probability is divided

ChatStream Guide

PagedAttention for More Efficient LLM Serving

Hello, this is the Product Development Department at Qualiteg Inc. Today we would like to introduce PagedAttention, a technology that is indispensable for commercial LLM serving. Introduction PagedAttention is an extremely important technology for our company. By taking advantage of PagedAttention, you can improve GPU memory utilization during text generation

ChatStream Guide

[ChatStream] A ChatPrompt Class for meta-llama/Meta-Llama-3-8B-Instruct

In this post we introduce a ChatPrompt class* for Llama 3, which was released yesterday (April 19, 2024). from chatstream import AbstractChatPrompt SYSTEM_PROMPT = """\ You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any

ChatStream Guide

Build a Chat Application with RakutenAI-7B-chat in 5 Minutes

Hello, this is the Product Development Department at Qualiteg Inc. Today we will build a full-fledged chat application using RakutenAI-7B-chat and ChatStream 0.7.0. RakutenAI-7B-chat is a model created by continually pre-training Mistral 7B on Japanese and then chat-tuning it. It ranks near the top of the Japanese LLM

ChatStream Guide

[ChatStream] ChatPrompt for Rakuten/RakutenAI-7B-chat

Here is the ChatPrompt for Rakuten/RakutenAI-7B-chat, which was released yesterday. from chatstream import AbstractChatPrompt SYSTEM_PROMPT = """\ A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. \ """ class ChatPromptRakutenMistral(

ChatStream Guide

[ChatStream] ChatPrompt for lightblue/karasu-7B-chat-plus

Hello, this is the Product Development Team at Qualiteg. In this article, we introduce the ChatPrompt for lightblue/karasu-7B-chat-plus. Customers using an older version of ChatStream can use this model by importing this ChatPrompt. (It is bundled in the latest release.) from chatstream import AbstractChatPrompt from chatstream.chat_prompt.role_

ChatStream Guide

[ChatStream] ChatPrompt for Llama 2

Hello, this is the Product Development Department at Qualiteg. In this article, we introduce the Llama 2-compatible ChatPrompt that is now bundled with ChatStream. The current ChatPrompt is shown below. If you are using an older version of ChatStream, you can add Llama 2 support with the following code. (Naturally,

ChatStream Guide

[ChatStream] ChatPrompt for rinna/nekomata-14b-instruction

In this article we introduce a ChatPrompt for rinna/nekomata-14b-instruction, which was released on December 21, 2023. The nekomata series is based on Qwen and has a vocabulary size of 152,000, considerably larger than earlier models, so we have high expectations for its Japanese language capabilities. This time we

ChatStream Guide

[ChatStream] Adding a Progress Bar to Slow Model Loading

Hello, this is the Product Development Division at Qualiteg. Loading an LLM from Hugging Face can take a very long time. In this article, we introduce a handy tool for exactly that situation. When you download a Hugging Face LLM, you can see the progress, but once the model has

ChatStream Guide

[ChatStream] Transformer Mock: Mocking Transformer Responses

Hello from the Product Development Team at Qualiteg. In this article we explain how to create mock data. This is for a feature formally called "Transformer Mock", which records actual LLM output and replays it later. Why would you need such a thing? It is used for testing

ChatStream Guide

[ChatStream] Generator Mock Responses for When LLM Loading Takes Too Long

Hello from the Product Development Team at Qualiteg. In this article, we introduce a technique you can use when loading an LLM takes a long time. For testing, or when implementing a sample app that uses the ChatStream API, there are situations where you want to restart ChatStream several times

ChatStream Guide

[ChatStream] Building a Console Chat

Hello from the Product Development Department at Qualiteg Inc. In this article, we explain how to build a console chat using ChatStream. Using the handle_console_input method, you can easily create a CLI-based chat and try out a model. chat.py import asyncio import torch from transformers import AutoTokenizer,

ChatStream Guide

[ChatStream] Starting the Web Server (ASGI Server)

Hello from the Product Development Department at Qualiteg Inc. In this article, we explain how to start a web server with ChatStream on board. uvicorn (started from within the code) Because ChatStream supports FastAPI/Starlette, it can be run on an ASGI server. To define uvicorn within your code, implement

ChatStream Guide

[ChatStream] The Queueing System and Concurrency Limits

Hello from the Product Development Department at Qualiteg Inc. In this article, we explain the queueing system in ChatStream. What Is the Queueing System? When ChatStream receives a large number of simultaneous access requests, it can queue those requests and limit the number of text generations that run at the