NumPy/PyTorch
This series is for machine learning engineers just getting started: it helps you build a clear mental picture of the tensor operations you use all the time in PyTorch and NumPy. Rather than a reference that aims for strict formal rigor, we will learn through the kinds of code you
LLM Security
Introduction With the rapid evolution of artificial intelligence, large language models (LLMs) have become indispensable tools for many companies and organizations. Their range of applications—natural language processing, code generation, data analysis, and more—expands daily, driving innovation across a wide range of fields, from streamlining business processes to supporting
Daily Dev Tips
Hello! When working with ONNX Runtime, you may run into an error like the following [E:onnxruntime:Default, provider_bridge_ort.cc:1744 TryGetProviderInfo_CUDA] Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory [W:onnxruntime:
![[LLM Security] How to Detect Hallucinations](https://images.unsplash.com/photo-1488272690691-2636704d6000?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3wxMTc3M3wwfDF8c2VhcmNofDI5fHxoYWNrZXJ8ZW58MHx8fHwxNzIyOTA1NTE5fDA&ixlib=rb-4.0.3&q=80&w=600)
LLM Security
Hello, this is the Qualiteg Research Team. Today, we walk through the following paper on hallucination detection in RAG systems and share our thoughts on LLMs built specifically to detect hallucinations. "Lynx: An Open Source Hallucination Evaluation Model" https://arxiv.org/pdf/2407.08488 Overview The paper describes
LLM
On July 23, 2024, Meta announced its latest large language models, the Llama 3.1 series. In this article, we introduce the features and performance of the Llama 3.1 series, along with a hands-on look at using it. We have also covered this topic in the video below, so
LLM
Today we tried out "mistralai/mamba-codestral-7B-v0.1" (commonly known as Codestral Mamba 7B), a new code-generation LLM released on July 16, 2024. This model adopts the new Mamba architecture and is published under the Apache 2 license. Performance Approaching SOTA Code-Generation Models Although Codestral 7B is built on
IT & AI Technology
Hello, this is the Product Development Department at Qualiteg. When a new model is released, we often go hunting for the right recipe to speed up inference — quantizing with various techniques and switching between multiple inference engines. In the course of that work, we ran into the following issue. Since
ChatStream
Hello, this is the Product Development Department at Qualiteg. Today we present a first-look report on CyberAgentLM3-22B-Chat (cyberagent/calm3-22b-chat), the latest LLM from CyberAgent, announced in yesterday's press release. Demo You can actually try chatting with calm3-22b-chat at the following site: https://chatstream.net/?ws_name=chat_app&
ChatStream
Hello! Today we tried out Llama-3-Elyza-JP-8B. Announced just yesterday, on June 26, 2024 (https://prtimes.jp/main/html/rd/p/000000046.000047565.html), Llama-3-Elyza-JP-8B is a Japanese LLM whose 70B version is reported to outperform GPT-4. In this post, we share our experience loading Llama-3-Elyza-JP-8B into our own Playground environment
IT & AI Technology
Hello, this is the Qualiteg Research Team. What Is Speculative Decoding? Speculative decoding is a technique for accelerating inference in large language models (LLMs). It has been reported to speed up most models by roughly 1.4x to 2.0x. In this approach, a small model (the draft model) produces
DIY GPU Machines
Without further ado, let's run some Python code on the GPU. 4.3 Writing GPU-Accelerated PyTorch Code in Jupyter Notebook STEP 1: Open a terminal and create a directory for the PyTorch project Enter the following commands: mkdir pytorch_pj cd pytorch_pj STEP 2: Launch Jupyter Notebook
AI Mathematics
Good morning! This is the Qualiteg Research Team. In this article, we explain how to express the cross-entropy computation using vectors and matrices. Chapter 8: Cross-Entropy and Vector Operations First of all, why would we want to express cross-entropy with vectors and matrices? The reason is that when you actually
ChatStream Guide
[Appendix] Comparison of Penalties Penalty type Purpose How it is applied Example penalty Repetition Penalty Prevents specific tokens or phrases from being repeated. Applies a penalty to the log probabilities (logits) of all previously generated tokens. For example, if a token has already been generated, its log probability is divided
IT & AI Technology
Hello! This is the Qualiteg Tech Blog. [Updated July 2, 2024] Today we will cover the features and pricing of the Google Gemini 1.5 Pro / Gemini 1.5 Flash models, and how to generate text with them using Python. What is Google Gemini? Google Gemini is a generative AI
DIY GPU Machines
In this installment, we install CUDA and PyTorch. 4.2 Installing CUDA (+cuDNN) and PyTorch Together Installing PyTorch also pulls in CUDA and cuDNN for you, so let's take advantage of that. STEP 1: Generate the PyTorch installation command With all the groundwork finally in place, it'
Daily Dev Tips
Hello from the Product Development Team at Qualiteg. TensorRT-LLM is an inference engine from NVIDIA that can be regarded as the successor to FasterTransformer, and it is available as one of the inference engines for our ChatStream. Like vLLM, it adds support for new models quickly and already supports a