NumPy/PyTorch
Introduction Hello! This is the Qualiteg Product Development Team. At Qualiteg, we research and develop AI characters and AI humans built on LLM technology. When reproducing virtual humans with "human-like" qualities such as facial expressions and gestures, AI-powered image processing—image generation and image editing—becomes essential. AI
Daily Dev Tips
Hello! When working with ONNX Runtime, you may run into an error like the following [E:onnxruntime:Default, provider_bridge_ort.cc:1744 TryGetProviderInfo_CUDA] Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory [W:onnxruntime:
Blog
Hello! Today we take a look at domain-specific LLMs. In recent years, large language models (LLMs) have advanced at a remarkable pace. While general-purpose AI such as GPT-4o and Claude 3.5 draws much of the attention, demand for LLMs specialized for particular fields and use cases is growing rapidly.
IT & AI Technology
Hello, this is the Product Development Department at Qualiteg. When a new model is released, we often go hunting for the right recipe to speed up inference — quantizing with various techniques and switching between multiple inference engines. In the course of that work, we ran into the following issue. Since
Daily Dev Tips
This is a list of NVIDIA GPU Compute Capability levels. The various acceleration features supported by inference engines depend on the Compute Capability level, since the hardware accelerators and dedicated functions available differ by level. Data Center / Professional GPUs GeForce GPU Capability Level Architecture - GeForce RTX 5090 120 Blackwell
IT & AI Technology
Hello, this is the Qualiteg Research Team. What Is Speculative Decoding? Speculative decoding is a technique for accelerating inference in large language models (LLMs). It has been reported to speed up most models by roughly 1.4x to 2.0x. In this approach, a small model (the draft model) produces
DIY GPU Machines
Without further ado, let's run some Python code on the GPU. 4.3 Writing GPU-Accelerated PyTorch Code in Jupyter Notebook STEP 1: Open a terminal and create a directory for the PyTorch project Enter the following commands: mkdir pytorch_pj cd pytorch_pj STEP 2: Launch Jupyter Notebook
DIY GPU Machines
In this installment, we install CUDA and PyTorch. 4.2 Installing CUDA (+cuDNN) and PyTorch Together Installing PyTorch also pulls in CUDA and cuDNN for you, so let's take advantage of that. STEP 1: Generate the PyTorch installation command With all the groundwork finally in place, it'
Daily Dev Tips
Hello from the Product Development Team at Qualiteg. TensorRT-LLM is an inference engine from NVIDIA that can be regarded as the successor to FasterTransformer, and it is available as one of the inference engines for our ChatStream. Like vLLM, it adds support for new models quickly and already supports a
DIY GPU Machines
In this installment, we install Anaconda. Python applications are generally run inside a virtual environment created for each application. Two tools are commonly used to build such virtual environments: Anaconda, or Python's built-in venv. This time, we will go with Anaconda. 4.1 Installing Anaconda (the Python environment)
ChatStream Guide
Hello, this is the Product Development Department at Qualiteg Inc. Today we would like to introduce PagedAttention, a technology that is indispensable for commercial LLM serving. Introduction PagedAttention is an extremely important technology for our company. By taking advantage of PagedAttention, you can improve GPU memory utilization during text generation
News
On April 25, 2024, Qualiteg Inc. was accepted into the NVIDIA Inception program. This marks a new step forward for us as we join innovative startups from around the world in developing and bringing advanced technologies to market. What This Means for Our LLM Service Development Being accepted into the
DIY GPU Machines
Hello! In this installment, we install the NVIDIA display driver so that we can use the GPU on Ubuntu. 3.2 Installing the NVIDIA Display Driver We will install the display driver that supports the GPU we purchased, an NVIDIA RTX 3090 Ti 24GB. 3.2.1 Downloading the Driver
DIY GPU Machines
Hello! In this installment, we configure Ubuntu after the OS installation. 3.1 Configuring Ubuntu 3.1.1 Updates on first boot At this point the Ubuntu OS installation is complete. If any updates are shown, such as an Ubuntu OS update or a language pack update, run those first.
DIY GPU Machines
Hello! Let's install Ubuntu 22.04 on the GPU machine from the USB drive we created last time. 2.0 Connect the LAN cable Before installing, connect a LAN cable to the GPU machine so that it has internet access. 2.1 Boot from the USB drive Insert
DIY GPU Machines
Hello! Starting with this installment, we move on to the software side of turning our GPU machine into an AI machine. The goal is to get the GPU working under Ubuntu and, ultimately, to reach the point where we can run inference with an LLM. 1. Creating a Ubuntu Installation