AI Math Basics: Logarithms and Exponentials, Part 2: Differentiating Logarithmic Functions (Introduction)

Good morning from the Qualiteg Research Team.

Starting today, we will spend two articles learning about logarithmic functions.
To be precise, we will learn how to differentiate logarithmic functions.

What was a logarithmic function again?

First of all, a logarithm is a way of finding "how many times a number \( b \) must be multiplied by itself to obtain another number \( x \)". For example, if \( b \) is 2 and \( x \) is 8, multiplying 2 by itself 3 times gives 8. In this case, mathematically we say "the logarithm of \( x \) to the base \( b \)".

Expressed as a formula, it looks like this:

$$
\log_b(x) = y
$$

Here \( b \) is the base, \( x \) is the argument (or antilogarithm), and \( y \) is the number of times \( b \) must be multiplied by itself to obtain \( x \). For example, when \( b = 2 \) and \( x = 8 \), we get \( y = 3 \), because multiplying 2 by itself 3 times gives 8.

$$
\log_2(8) = 3
$$

A logarithmic function can also be thought of as an "inverse" function. That is, just as division is the inverse of multiplication, the logarithm is the inverse of exponentiation (for example, 2 to the power of 3 is 8).

Why are logarithmic functions important?

First, let us explain why we need to learn about logarithmic functions at all.

Logarithmic functions appear in the formula for the cross-entropy function, the representative loss function for classification problems in machine learning.

Cross-entropy function for multi-class classification

$$
\ - \frac{1}{N} \sum_{i}^{N} \sum_{k}^{K} t_{ik} \log y_{ik} \tag{1}
$$

Cross-entropy function for binary classification

$$
\ - \frac{1}{N} \sum_{i}^{N} \lbrack t_{i} \log y_{i} + (1- t_{i}) \log (1- y_{i}) \rbrack \tag{2}
$$

When training a neural network, we train it so as to minimize the loss function. To find the direction (gradient) in which the loss function decreases, we compute the derivative of the loss function.

Since the cross-entropy function contains a logarithmic function, as shown above, we need to be able to differentiate logarithmic functions.

The goal of this series is to master the differentiation of logarithmic functions. To understand how to differentiate logarithmic functions, we will first master the fundamentals of logarithmic functions and then use them to learn differentiation.

1. What is the logarithmic function \(log\)?

Now, let us explain logarithmic functions once more, this time in terms of formulas.

$$
y = \log_a x
$$

The logarithm \(\log\) finds the power to which \(a\) must be raised to obtain \(x\). This is called the logarithmic function with base \(a\).

When \(y=\log_a x\), we have \(a^{y} = x\).

Also, \(y=\log_a x\) is the inverse function of \(y=a^{x}\).

Example

With \(y=\log_2 x\), setting \(x=8\) gives \(y=\log_2 8\), i.e., we are asking to what power \(2\) must be raised to obtain \(8\), so the answer is \(y=3\).

How was that?

Looking back on it (or, if this was your first time, reflecting on what you just learned), it is surprisingly approachable, isn't it?

See you next time!

navigation