Cross-Entropy Explained, Part 1: Two Formulas for One Loss Function

Good morning from the Qualiteg Research Department.

Starting today, we would like to take a thorough look at cross-entropy.

Are there two formulas for the cross-entropy function?

In this series, we take up the cross-entropy function, which is commonly used as the loss function for classification problems in machine learning.

In fact, the differentiation of exponential and logarithmic functions that we have studied so far was all in preparation for understanding this cross-entropy function in depth.

Understanding what properties cross-entropy has and how it is derived will also be a great help later on when we unravel how LLMs work, so we want to examine it carefully.

So, let's get right to it.

Equation \((1)\) below is the cross-entropy function.

$$
\ - \frac{1}{N} \sum_{i}^{N} \sum_{k}^{K} t_{ik} \log y_{ik} \tag{1}
$$

Equation \((2)\) below is also the cross-entropy function.

$$
\ - \frac{1}{N} \sum_{i}^{N} \lbrack t_{i} \log y_{i} + (1- t_{i}) \log (1- y_{i}) \rbrack \tag{2}
$$

If you search for "cross-entropy function," these two equations are what you will usually find.

A natural question comes to mind: "Wait, are there two definitions?" In fact, both can be derived from the same starting point.

Rather than simply memorizing the formulas, we want to internalize what they really are, so we will expand the derivation step by step without skipping anything and build a solid understanding of the concept.

That means we will look at and think about similar things repeatedly, and perhaps a little insistently.

To give away the ending in advance: equation \((1)\) is the cross-entropy for multi-class classification (multi-class classification = the task of assigning input data to one of several classes), while equation \((2)\) is the cross-entropy for binary classification, used for binary classification (the task of assigning input data to one of two classes).

The binary cross-entropy function in equation \((2)\) is what you get by expanding the multi-class cross-entropy in equation \((1)\) for the special case of binary classification, and it can be derived easily from equation \((1)\).
We will explain that expansion in detail later as well.

What we want to understand in this series

• Resolve the simple question: "There seem to be two cross-entropy functions, so which one is correct?"

• Understand the differences between cross-entropy, categorical cross-entropy (for multi-class problems), and binary cross-entropy (for binary problems), and when to use each

• Understand what cross-entropy actually is in the first place, and where it is derived from

Chapter 1: Cross-entropy in classification problems

There is a wide variety of loss functions used in neural networks, but for classification problems, the standard choice is undoubtedly the cross-entropy loss function.

It is implemented in every framework, and whenever you decide to "give deep learning a try," you will inevitably rely on it.

Classification problems can be broadly divided into two kinds.

• Binary classification (two-class classification)
  • Predicts which of two classes the input data belongs to.
    It is also called two-class classification.
    Binary classification can predict whether the answer is "yes" or "no."
    The famous data science tutorial "Did a passenger on the Titanic survive or not?" is also a binary classification problem.

[Examples of binary classification]

• "Is this email spam or not?"
• "Will this customer buy or not?"
• "Will this student pass or not?"
• "Is this movie review positive or negative?"

• Multi-class classification
  • Predicts which of several classes the input data belongs to. Unlike binary classification, there may be any number of classes, but the number of classes to classify into is decided in advance.
    For example, predicting which of five classes an input image belongs to:
    "dog," "fox," "wolf," "cat," or "raccoon dog."

The loss function used when training these binary and multi-class classifiers is the cross-entropy loss function.

We have separated multi-class classification from binary (two-class) classification, but logically speaking, multi-class classification, as the name suggests, predicts which of several classes the input data belongs to, so binary classification is included within multi-class classification. If we let the number of classes be \(k\), then the case \(k=2\) is binary classification, and \(k>2\) is what we usually call multi-class classification.

So logically, binary classification can be regarded as a special case of multi-class classification, yet at first glance a different cross-entropy loss function is used, as shown below.

Why is that?

We will work carefully through the equations, including this question, so that cross-entropy ultimately becomes clear.

The cross-entropy loss function for binary classification is called Binary Cross Entropy, and deep learning frameworks implement it under the abbreviation \(BCE\).
(Binary means two-valued, two-component, or two-way. Since it distinguishes input data into two categories, "A or not A," the name fits perfectly.)

$$
\ - \frac{1}{N} \sum_{i}^{N} \lbrack t_{i} \log y_{i} + (1- t_{i}) \log (1- y_{i}) \rbrack \tag{2, repeated}
$$

The cross-entropy loss function for multi-class classification is called Categorical Cross Entropy or Multi-Class Cross Entropy.

$$
\ - \frac{1}{N} \sum_{i}^{N} \sum_{k}^{K} t_{ik} \log y_{ik} \tag{1, repeated}
$$

So, in this installment we have surveyed the kinds of problems in which the cross-entropy loss function plays a role.

Next time, we will look at the essence of classification problems and the likelihood function.

See you in the next installment.

References
https://journal.qualiteg.com/books/

navigation