![[Updated May 14, 2024] LLM Inference API Pricing and Inference Speed](/content/images/size/w600/2024/05/LLM--------1.png)
IT & AI Technology
Here is a summary of the pay-as-you-go pricing and generation speeds for using LLMs via API. We plan to update it over time. [API Pricing] shows the output-side price per 1 million tokens. [Generation Speed] is expressed in "tokens/s" (tokens per second), i.e., how many tokens
AI Mathematics
Good morning from the Qualiteg Research Team. This time, let's look at cross-entropy for binary classification. Chapter 7: Cross-Entropy for Binary Classification 7-1. Cross-Entropy for Binary Classification (Single-Sample Version) From here, we would like to derive the cross-entropy for binary classification. Binary classification predicts which of two classes
![[ChatStream] Pin the Revision to Guard Against Unexpected Changes in Input/Output Prompts](/content/images/size/w600/2024/05/-----TIPs--1--1.png)
Daily Dev Tips
Hello, this is the Product Development Department at Qualiteg Inc. During Japan's Golden Week holidays, the tokenizer.json of microsoft/Phi-3-mini-128k-instruct was changed. As a result, prompt parsing failed and chat streaming stopped working. Specifically, the following change was made: https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/commit/8a362e755d2faf8cec2bf98850ce2216023d178a Originally,
DIY GPU Machines
In this installment, we install Anaconda. Python applications are generally run inside a virtual environment created for each application. Two tools are commonly used to build such virtual environments: Anaconda, or Python's built-in venv. This time, we will go with Anaconda. 4.1 Installing Anaconda (the Python environment)
Daily Dev Tips
This post explains how to deal with the following error when you try to pip install bitsandbytes. Could not load bitsandbytes native library: libcusparse.so.11: cannot open shared object file: No such file or directory Traceback (most recent call last): File "/home/mlu/.virtualenvs/ChatStream/lib/python3.10/
ChatStream Guide
Hello, this is the Product Development Department at Qualiteg Inc. Today we would like to introduce PagedAttention, a technology that is indispensable for commercial LLM serving. Introduction PagedAttention is an extremely important technology for our company. By taking advantage of PagedAttention, you can improve GPU memory utilization during text generation
Daily Dev Tips
This post explains how to deal with the error "triu_tril_cuda_template" not implemented for 'BFloat16', which can occur when you specify torch_dtype=torch.bfloat16 while loading a model. Below is the log from when this occurred with Llama 3. File "/home/mlu/
![[ChatStream] A ChatPrompt Class for meta-llama/Meta-Llama-3-8B-Instruct](/content/images/size/w600/2024/04/ChatPrompt--4-.png)
ChatStream Guide
In this post we introduce a ChatPrompt class* for Llama 3, which was released yesterday (April 19, 2024). from chatstream import AbstractChatPrompt SYSTEM_PROMPT = """\ You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any
DIY GPU Machines
Hello! In this installment, we install the NVIDIA display driver so that we can use the GPU on Ubuntu. 3.2 Installing the NVIDIA Display Driver We will install the display driver that supports the GPU we purchased, an NVIDIA RTX 3090 Ti 24GB. 3.2.1 Downloading the Driver
AI Mathematics
Good morning from the Qualiteg Research Team. In this installment, we extend the previous discussion and derive the cross-entropy for multi-class classification with N samples, working through actual data as we go. Chapter 6: Cross-Entropy for Multi-Class Classification (N Samples) In real training, we perform batch learning: several training samples
Daily Dev Tips
Hello, this is the Qualiteg Product Development team. Overview * There have been reports that installing PyTorch, which ChatStream requires, does not work properly on Python 3.12. * Installation succeeded on Python 3.11 and earlier. Error message pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 Running
DIY GPU Machines
Hello! In this installment, we configure Ubuntu after the OS installation. 3.1 Configuring Ubuntu 3.1.1 Updates on first boot At this point the Ubuntu OS installation is complete. If any updates are shown, such as an Ubuntu OS update or a language pack update, run those first.
ChatStream Guide
Hello, this is the Product Development Department at Qualiteg Inc. Today we will build a full-fledged chat application using RakutenAI-7B-chat and ChatStream 0.7.0. RakutenAI-7B-chat is a model created by continually pre-training Mistral 7B on Japanese and then chat-tuning it. It ranks near the top of the Japanese LLM
AI Mathematics
Good morning! This is the Research Department at Qualiteg Inc. In this installment, we will calculate the cross-entropy used for multi-class classification. Chapter 5: Calculating Cross-Entropy for Multi-Class Classification (Single-Sample Version) First, let us restate the cross-entropy function (single-sample version). $$ \ - \log L=\sum_{k=1}^{K} t_{k} \log
![[ChatStream] ChatPrompt for Rakuten/RakutenAI-7B-chat](/content/images/size/w600/2024/04/ChatPrompt--2-.png)
ChatStream Guide
Here is the ChatPrompt for Rakuten/RakutenAI-7B-chat, which was released yesterday. from chatstream import AbstractChatPrompt SYSTEM_PROMPT = """\ A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. \ """ class ChatPromptRakutenMistral(
DIY GPU Machines
Hello! Let's install Ubuntu 22.04 on the GPU machine from the USB drive we created last time. 2.0 Connect the LAN cable Before installing, connect a LAN cable to the GPU machine so that it has internet access. 2.1 Boot from the USB drive Insert