IT & AI Technology
Hello, this is the Product Development Department at Qualiteg. When a new model is released, we often go hunting for the right recipe to speed up inference — quantizing with various techniques and switching between multiple inference engines. In the course of that work, we ran into the following issue. Since
ChatStream
Hello, this is the Product Development Department at Qualiteg. Today we present a first-look report on CyberAgentLM3-22B-Chat (cyberagent/calm3-22b-chat), the latest LLM from CyberAgent, announced in yesterday's press release. Demo You can actually try chatting with calm3-22b-chat at the following site: https://chatstream.net/?ws_name=chat_app&
Daily Dev Tips
When you encounter an error like the following: RuntimeError: implement_array_function method already has a docstring This error message appears to stem from a problem inside the numpy package. Specifically, the error says that the implement_array_function method already has documentation attached, and it typically seems to occur
ChatStream
Hello! Today we tried out Llama-3-Elyza-JP-8B. Announced just yesterday, on June 26, 2024 (https://prtimes.jp/main/html/rd/p/000000046.000047565.html), Llama-3-Elyza-JP-8B is a Japanese LLM whose 70B version is reported to outperform GPT-4. In this post, we share our experience loading Llama-3-Elyza-JP-8B into our own Playground environment
Daily Dev Tips
How to resolve the following error File "/venv/Lib/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/venv/Lib/site-packages/transformers/generation/utils.py", line 1744, in generate model_kwargs["past_key_values"] = self._get_cache( ^^^^^^^^^^^^^^^^ File
Daily Dev Tips
This is a list of NVIDIA GPU Compute Capability levels. The various acceleration features supported by inference engines depend on the Compute Capability level, since the hardware accelerators and dedicated functions available differ by level. Data Center / Professional GPUs GeForce GPU Capability Level Architecture - GeForce RTX 5090 120 Blackwell
Daily Dev Tips
In this article, we introduce sample code for using the OpenAI API. The OpenAI API is not limited to OpenAI's own service; other inference engines such as vLLM also expose OpenAI-compatible API servers, and it is becoming the de facto standard for LLM serving APIs. So it is
NumPy/PyTorch
Optimizing PyTorch Tensor Operations: Understanding and Resolving a Warning Message Hello! This is the Qualiteg Product Development Team. While working with PyTorch 1.13, we ran into the following warning message: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach(
![[ChatStream] Quickly Measuring Inference Speed Under Concurrent Requests](/content/images/size/w600/2024/05/--------------2-.png)
Hello, this is the Qualiteg Product Development Team. This time, we ran a quick measurement of RakutenAI-7B-Chat's performance when handling five simultaneous requests. Test Environment * GPU: A5000 * LLM: RakutenAI-7B-Chat AWQ-8 * Inference environment: ChatStream v0.7.2 * Number of concurrent requests: 5 Since a video is probably easier to
ChatStream Guide
[Appendix] Comparison of Penalties Penalty type Purpose How it is applied Example penalty Repetition Penalty Prevents specific tokens or phrases from being repeated. Applies a penalty to the log probabilities (logits) of all previously generated tokens. For example, if a token has already been generated, its log probability is divided
Daily Dev Tips
Hello from the Product Development Team at Qualiteg. TensorRT-LLM is an inference engine from NVIDIA that can be regarded as the successor to FasterTransformer, and it is available as one of the inference engines for our ChatStream. Like vLLM, it adds support for new models quickly and already supports a
![[Updated May 14, 2024] LLM Inference API Pricing and Inference Speed](/content/images/size/w600/2024/05/LLM--------1.png)
IT & AI Technology
Here is a summary of the pay-as-you-go pricing and generation speeds for using LLMs via API. We plan to update it over time. [API Pricing] shows the output-side price per 1 million tokens. [Generation Speed] is expressed in "tokens/s" (tokens per second), i.e., how many tokens
![[ChatStream] Pin the Revision to Guard Against Unexpected Changes in Input/Output Prompts](/content/images/size/w600/2024/05/-----TIPs--1--1.png)
Daily Dev Tips
Hello, this is the Product Development Department at Qualiteg Inc. During Japan's Golden Week holidays, the tokenizer.json of microsoft/Phi-3-mini-128k-instruct was changed. As a result, prompt parsing failed and chat streaming stopped working. Specifically, the following change was made: https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/commit/8a362e755d2faf8cec2bf98850ce2216023d178a Originally,
Daily Dev Tips
This post explains how to deal with the following error when you try to pip install bitsandbytes. Could not load bitsandbytes native library: libcusparse.so.11: cannot open shared object file: No such file or directory Traceback (most recent call last): File "/home/mlu/.virtualenvs/ChatStream/lib/python3.10/
ChatStream Guide
Hello, this is the Product Development Department at Qualiteg Inc. Today we would like to introduce PagedAttention, a technology that is indispensable for commercial LLM serving. Introduction PagedAttention is an extremely important technology for our company. By taking advantage of PagedAttention, you can improve GPU memory utilization during text generation
Daily Dev Tips
This post explains how to deal with the error "triu_tril_cuda_template" not implemented for 'BFloat16', which can occur when you specify torch_dtype=torch.bfloat16 while loading a model. Below is the log from when this occurred with Llama 3. File "/home/mlu/