AI Math Basics: Does AI Math Dream of Minecraft?

Hello everyone, this is the Research Department at Qualiteg.

When reading papers on LLMs, do you ever feel dizzy from the sheer number of mysterious formulas and symbols?

I studied neural networks back in my student days, and I remember being taken aback by how the mathematical background needed to understand them was layered like a mille-feuille.

Compared with the perceptron and early neural network research, or even with the analytical deep-learning-based AI of more recent years, modern LLMs have a mille-feuille roughly ten times thicker. To understand mathematically how an LLM works, you have to grasp the many, sometimes hundreds of, layers of mathematical theory underneath.

(If you only use LLMs, you are essentially eating the whole mille-feuille and saying "delicious!" But building that multi-layered mille-feuille (of mathematics) yourself, understanding it one layer at a time, is a very different level of effort from simply eating it.)

As you can see, LLMs are built on a mathematical body of theory accumulated through the tireless efforts of many predecessors and pioneers: probability and statistics as the foundation of machine learning, classical machine learning theory, the calculus and discretization used in deep learning, then RNNs and LSTMs, on to the Transformer and attention, then BERT, and from BERT to what we now call GPT.

Such a body of theory cannot be mastered overnight, but after several years of study, it seems to me that there is a common set of important mathematical fundamentals.

So we thought it would be worthwhile to explain those mathematical foundations in an accessible way here on the Qualiteg blog, and we are starting a series of posts under the title "AI Math".

Roughly speaking, then, the goal of this series is to "get a hands-on feel for the formulas that appear in arXiv papers".

Once you get used to formulas and mathematics, they start to look like blocks in Minecraft. It is a feeling close to: place a sticky piston, a repeater, and a comparator like this, wire up the redstone circuit like that, and you get a flip-flop. (Or maybe that is just me.)

If, after reading this series, formulas and equations start to look like Minecraft blocks to you, we will be delighted 😄

navigation