Qualiteg - Qualiteg Product Development Team

ChatStream Guide

[ChatStream] A ChatPrompt Class for meta-llama/Meta-Llama-3-8B-Instruct

In this post we introduce a ChatPrompt class* for Llama 3, which was released yesterday (April 19, 2024). from chatstream import AbstractChatPrompt SYSTEM_PROMPT = """\ You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any

Daily Dev Tips

How to Fix a Failed PyTorch Installation on Python 3.12

Hello, this is the Qualiteg Product Development team. Overview * There have been reports that installing PyTorch, which ChatStream requires, does not work properly on Python 3.12. * Installation succeeded on Python 3.11 and earlier. Error message pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 Running

ChatStream Guide

Build a Chat Application with RakutenAI-7B-chat in 5 Minutes

Hello, this is the Product Development Department at Qualiteg Inc. Today we will build a full-fledged chat application using RakutenAI-7B-chat and ChatStream 0.7.0. RakutenAI-7B-chat is a model created by continually pre-training Mistral 7B on Japanese and then chat-tuning it. It ranks near the top of the Japanese LLM

ChatStream Guide

[ChatStream] ChatPrompt for Rakuten/RakutenAI-7B-chat

Here is the ChatPrompt for Rakuten/RakutenAI-7B-chat, which was released yesterday. from chatstream import AbstractChatPrompt SYSTEM_PROMPT = """\ A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. \ """ class ChatPromptRakutenMistral(

Daily Dev Tips

Packing LLM Data Files into an Uncompressed ZIP

When building LLM services, we often need to move an LLM that has already been downloaded once to a different server. (When you want to use the same LLM model on servers with an identical configuration, this is faster than downloading it again from Hugging Face, for example.) When a

Daily Dev Tips

Buffering Incrementally Generated Tokens

Hello from the Qualiteg Product Development Department. Today we would like to introduce a way to deal with the fragmented-token problem that often comes up during inference. In the incremental generation used for streaming chat, text is generated one token at a time. A token, however, is neither a "

Non-LLM Generative AI

Trying Out MagicAnimate

Magic Animate is an open-source project that turns still images into videos. In this article, we try it out using the Windows version. The test environment is a Core i7 + 64 GB RAM + RTX 3090. Note that installation takes about 30 minutes and consumes roughly 32 GB of storage. Creating

ChatStream Guide

[ChatStream] ChatPrompt for lightblue/karasu-7B-chat-plus

Hello, this is the Product Development Team at Qualiteg. In this article, we introduce the ChatPrompt for lightblue/karasu-7B-chat-plus. Customers using an older version of ChatStream can use this model by importing this ChatPrompt. (It is bundled in the latest release.) from chatstream import AbstractChatPrompt from chatstream.chat_prompt.role_

ChatStream Guide

[ChatStream] ChatPrompt for Llama 2

Hello, this is the Product Development Department at Qualiteg. In this article, we introduce the Llama 2-compatible ChatPrompt that is now bundled with ChatStream. The current ChatPrompt is shown below. If you are using an older version of ChatStream, you can add Llama 2 support with the following code. (Naturally,

ChatStream Guide

[ChatStream] ChatPrompt for rinna/nekomata-14b-instruction

In this article we introduce a ChatPrompt for rinna/nekomata-14b-instruction, which was released on December 21, 2023. The nekomata series is based on Qwen and has a vocabulary size of 152,000, considerably larger than earlier models, so we have high expectations for its Japanese language capabilities. This time we

ChatStream Guide

[ChatStream] Adding a Progress Bar to Slow Model Loading

Hello, this is the Product Development Division at Qualiteg. Loading an LLM from Hugging Face can take a very long time. In this article, we introduce a handy tool for exactly that situation. When you download a Hugging Face LLM, you can see the progress, but once the model has

ChatStream Guide

[ChatStream] Transformer Mock: Mocking Transformer Responses

Hello from the Product Development Team at Qualiteg. In this article we explain how to create mock data. This is for a feature formally called "Transformer Mock", which records actual LLM output and replays it later. Why would you need such a thing? It is used for testing

ChatStream Guide

[ChatStream] Generator Mock Responses for When LLM Loading Takes Too Long

Hello from the Product Development Team at Qualiteg. In this article, we introduce a technique you can use when loading an LLM takes a long time. For testing, or when implementing a sample app that uses the ChatStream API, there are situations where you want to restart ChatStream several times

ChatStream Guide

[ChatStream] Building a Console Chat

Hello from the Product Development Department at Qualiteg Inc. In this article, we explain how to build a console chat using ChatStream. Using the handle_console_input method, you can easily create a CLI-based chat and try out a model. chat.py import asyncio import torch from transformers import AutoTokenizer,

ChatStream Guide

[ChatStream] Starting the Web Server (ASGI Server)

Hello from the Product Development Department at Qualiteg Inc. In this article, we explain how to start a web server with ChatStream on board. uvicorn (started from within the code) Because ChatStream supports FastAPI/Starlette, it can be run on an ASGI server. To define uvicorn within your code, implement

ChatStream Guide

[ChatStream] The Queueing System and Concurrency Limits

Hello from the Product Development Department at Qualiteg Inc. In this article, we explain the queueing system in ChatStream. What Is the Queueing System? When ChatStream receives a large number of simultaneous access requests, it can queue those requests and limit the number of text generations that run at the