Cross-Entropy Explained, Part 3: Deriving Cross-Entropy from the Likelihood Function

Good morning! This is the Qualiteg Research Department.

In this installment, we will derive cross-entropy from the likelihood function.

Chapter 4: Deriving Cross-Entropy from the Likelihood Function

So far, we rolled a die 20 times, counted the events that occurred (whether the result was 1, 2, ..., or 6), and summarized the results in a table like the one below.

Now let's see what happens when we roll the die just once.

If the result of a single roll is a 1, we can write it as follows.
(We have simply placed a ✔ mark next to the face that came up.)

A ✔ mark is awkward to bring into calculations, so let's replace it with \(1\) for the face that came up and \(0\) for the faces that did not.

(It may help to think of it this way: for the face marked \(1\), frequency = probability = \(1\); for the faces marked \(0\), they did not appear in that trial, so frequency = probability = \(0\).)

The Result column can then be expressed with \(1\) and \(0\) as follows.

Furthermore, as in the earlier tables, if we again express the Result column as the frequency with which the event occurs, denoted \(t\), we get the following.

Writing this once more as the log-likelihood function gives

$$
\log L=\sum_{k=1}^{6} t_{k} \log y_{k} \tag{4.1}
$$

Yes, this equation \((4.1)\) is exactly the same as equation \((3.6)\). What differs from equation \((3.6)\) is the setting behind it: this one can be interpreted as the log-likelihood function for a single trial (*).

*Calling it the log-likelihood function for a single trial may sound more complicated than it is. If you think about it, a likelihood is fundamentally a product of multiple probabilities, so the log-likelihood function restricted to a single trial is not so much a likelihood as simply the logarithm of a probability written with an exponent; in substance, it is just a probability. So why bother with such a roundabout interpretation? Because it serves as a lead-in to the cross-entropy formula that appears later.

Now let's actually compute equation \((3.7)\) for a single trial in which the event "a 1 comes up" occurred.

$$
\begin{aligned}
\log L= &t_{1} \log y_{1} + t_{2} \log y_{2} + t_{3} \log y_{3} + t_{4} \log y_{4} + t_{5} \log y_{5} + t_{6} \log y_{6} &
\
= &1 \cdot \log y_{1} + 0 \cdot \log y_{2} + 0 \cdot \log y_{3} + 0 \cdot \log y_{4} + 0 \cdot \log y_{5} + 0 \cdot \log y_{6}
\end{aligned}
$$

Since only one of the frequencies \(t\) is \(1\) and the rest are \(0\), the calculation becomes this simple.

Here we dealt with a die, so there were 6 events, but if we generalize this to \(K\) events, we can write

$$
\log L=\sum_{k=1}^{K} t_{k} \log y_{k} \tag{4.2}
$$

This equation \((4.2)\) is the log-likelihood function per instance, or, in slightly more statistical language, the log-likelihood function per sample.

The larger the log-likelihood function, the more plausible the parameters \(y_{k}\) are. In machine learning such as deep learning, however, we train so that the loss function becomes smaller, so equation \((4.3)\), which is equation \((4.2)\) with a minus sign attached, is called the cross-entropy function.

Cross-entropy function (single-sample version)

$$
\ - \log L=\sum_{k=1}^{K} t_{k} \log y_{k} \tag{4.3}
$$

With that, we have derived the cross-entropy function.

And they all lived happily ever after 👏

What? Something's off?

Did we forget to tie up loose ends?

"What happened to the probabilities \(y\) of each face of the die?"

"Aren't we going to find the point where the derivative of the log-likelihood function is \(0\) and work out the probability of each face?"

"A point where the derivative of the log-likelihood function is \(0\) is a local maximum or minimum, not necessarily the global maximum, right?"

"Wait a moment. The reason we took the logarithm of the likelihood function was to make it easier to differentiate, wasn't it? Aren't we going to differentiate the log-likelihood function?"

Yes, you are absolutely right. We will address these points separately in a "Supplement."

The probability of each face of the die can indeed be found by maximum likelihood estimation, but the main focus of this series is deriving the "cross-entropy function." Now that "cross-entropy" is faintly coming into view, we would like to set the topic of estimating the die's probabilities aside for a little while and dig a bit deeper into cross-entropy. We hope you'll stay with us!

See you in the next installment!

References
https://journal.qualiteg.com/books/

navigation