Build a Chat Application with RakutenAI-7B-chat in 5 Minutes
Hello, this is the Product Development Department at Qualiteg Inc.
Today we will build a full-fledged chat application using RakutenAI-7B-chat and ChatStream 0.7.0.
RakutenAI-7B-chat is a model created by continually pre-training Mistral 7B on Japanese and then chat-tuning it. It ranks near the top of the Japanese LLM leaderboard https://wandb.ai/wandb-japan/llm-leaderboard/reports/Nejumi-LLM-Neo--Vmlldzo2MTkyMTU0, making it a highly promising model.
Source code
Without further ado, here is the source code.
Because the model is loaded with 4-bit quantization, it runs comfortably on a GPU such as an A4000 (16GB).
import logging
import torch
import uvicorn
from fastapi import FastAPI
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from chatstream import ChatStream, ChatPromptRakutenMistral as ChatPrompt, LoadTime, TokenSamplerIsok
from chatstream.fastersession.faster_session_rdb_store import FasterSessionRdbStore
from chatstream.util.session.chat_stream_session_on_set_value_listener import chat_stream_session_on_set_value_listener
"""
Sample ChatStream server program for 'Rakuten/RakutenAI-7B-chat'
- This is a development configuration (for production, we recommend scaling out with ChatStreamPool and placing a secure, robust load balancer / reverse proxy such as Qualiteg SunsetServer in front)
- Standalone mode (acts as a single-instance web application / Web API server, not as a node in scale-out mode)
- The model is handled with 4-bit quantization
"""
num_gpus = 1 # Number of GPUs used on this node
device = torch.device("cuda")
model_path = 'Rakuten/RakutenAI-7B-chat'
use_fast = True
# Config used for 4-bit quantization
quantization_config = BitsAndBytesConfig(load_in_4bit=True)
# Load the model (LoadTime shows a progress display while loading)
model = LoadTime(name=model_path, hf=True,
fn=lambda: AutoModelForCausalLM.from_pretrained(model_path,
quantization_config=quantization_config,
device_map="auto"))()
model.eval()
tokenizer = AutoTokenizer.from_pretrained(model_path)
# Configure the database used by ChatStream
database_def = {
"type": "rdbms", # "memory","rdbms","mongo","redis"
"rdbms": {
"db_url": "[your_db_url]" # Specify your DB here
},
"db": None # Specify a DB object directly here if you prefer
}
server_host_info = {'protocol': 'http', 'host': 'localhost', 'port': 9999}
num_of_concurrent_executions = 10 # Maximum number of concurrent text generations
max_new_tokens = 512 # Maximum number of tokens to generate
tokens_per_sec = 6 # Token generation speed (tokens/sec) when there are num_of_concurrent_executions concurrent connections
text_generation_timeout_sec = (max_new_tokens / tokens_per_sec) + 10 # Timeout per text generation. The +10 is a safety margin.
# Create the ChatStream instance
chat_stream = ChatStream(
num_of_concurrent_executions=num_of_concurrent_executions, # Maximum number of concurrent text generations
max_queue_size=5, # Size of the wait queue once the maximum number of generations is reached. Requests beyond this size receive a Too many requests error.
model_id='rakuten__rakuten_ai_7b_chat', # Model ID
model_support_languages=['ja', 'en'], # Languages supported by the model.
model_desc={
'disp_name': {'en': 'RakutenAI-7B-chat', 'ja': 'RakutenAI-7B-chat', }, # Model name shown in the UI
'about': {
'ja': '2024/3/21 にリリースされた Mistral-7B-v0.1 ベースの日本語LLM', # Description shown in the UI (Japanese)
'en': 'Japanese LLM based on Mistral-7B-v0.1 released on 3/21/2024', # Description shown in the UI (English)
},
'default_utterance_hints': [
{'utterance': {'ja': "Who starred in the movie 'Titanic' released in 1997?", # Sample utterance shown in the UI (Japanese)
'en': "Who starred in the movie 'Titanic' released in 1997?", # Sample utterance shown in the UI (English)
},
'desc': {'ja': '映画タイタニックの主演は？', # Description of the sample utterance shown in the UI (Japanese)
'en': '', # Description of the sample utterance shown in the UI (English)
}
},
]
},
server_host_info=server_host_info,
model=model,
tokenizer=tokenizer,
num_gpus=num_gpus,
device=device,
chat_prompt_clazz=ChatPrompt, # Set the ChatPrompt for this model
add_special_tokens=False, # Whether to add special tokens
text_generation_timeout_sec=text_generation_timeout_sec, # The timeout is calculated from the token generation speed at the maximum number of concurrent users x max_new_tokens
max_new_tokens=max_new_tokens, # Maximum number of tokens generated per request
context_len=1024, # Set the context length
temperature=0.7, # Set the sampling parameter temperature
top_k=10, # Sampling parameter top K
# top_p=0.9, # Set the sampling parameter top P
repetition_penalty=1.05, # Set the sampling parameter repetition penalty
database=database_def, # Set the database configuration
client_roles={
"user": {
"apis": {
"allow": "all", # [DefaultApiNames.CHAT_STREAM, ],
"auth_method": "nothing", # This ChatStream runs standalone, so no authentication is used (in scale-out mode, set appropriate server authentication)
"use_session": True, # This ChatStream runs standalone, so use_session:True. (When running in scale-out mode, set use_session:False)
}
},
}, # Set the roles.
locale='ja',
token_sampler=TokenSamplerIsok(), # TokenSamplerHft() # TokenSamplerIsok() #
seed=42,
)
chat_stream.logger.setLevel(logging.DEBUG)
# Store that persists session data to an RDBMS
rdb_store = FasterSessionRdbStore(database_def=database_def,
on_set_value_listener=chat_stream_session_on_set_value_listener, # Helper that skips non-serializable objects when persisting the session
)
# memory_store = get_chat_stream_session_memory_store() # Store session data in memory
# file_store = get_chat_stream_session_file_store() # Store session data in files
# Configuration dict for the middleware added to ChatStream
mw_opts = {
"faster_session": {
"secret_key": "chatstream-default-session-secret-key",
"store": rdb_store, # Store used to persist the session
# "same_site":"Strict", # same_site attribute of set-cookie, default is Strict
# "is_http_only":True,# http_only attribute of set-cookie, default is True
# "is_secure":True,# secure attribute of set-cookie, default is True
# "max_age":0 # max_age attribute of set-cookie, default is True
},
}
# Create the FastAPI instance
app = FastAPI()
# Automatically add the required middleware (it can also be configured manually)
chat_stream.append_middlewares(app, opts=mw_opts)
# Automatically add (mount) the required APIs
# All APIs are added here, but you can also select APIs according to your use case
# See default_api_paths.py for the details of each URL path
chat_stream.append_apis(app, {"all": True})
@app.on_event("startup")
async def startup():
# After the web server starts,
# start the ChatStream queueing system
await chat_stream.start_queue_worker()
def start_server():
# Start the web server
uvicorn.run(app, host=server_host_info.get('host'), port=server_host_info.get('port'))
def main():
start_server()
if __name__ == "__main__":
main()
For the ChatPrompt class used for model input and output, we use ChatPromptRakutenMistral, which ships with the latest version of ChatStream.
Alternatively, you can write your own by referring to the following article.
https://journal.qualiteg.com/chatprompt_rakuten_ai_7b_chat/
Now let's run this code and try chatting.
It started successfully, and we were able to chat with the model from a web browser.
At the Qualiteg Product Development Department, we promptly port each new model to ChatStream as soon as it is released on Hugging Face.
As a result, you can try even the newest models right away with almost no code. This time as well, we were able to implement a full-fledged LLM chat with little more than boilerplate.
See you next time with another LLM!
