Cross-Entropy Explained, Part 5: Cross-Entropy for Multi-Class Classification (N Samples)

Good morning from the Qualiteg Research Team.

In this installment, we extend the previous discussion and derive the cross-entropy for multi-class classification with N samples, working through actual data as we go.

Chapter 6: Cross-Entropy for Multi-Class Classification (N Samples)

In real training, we perform batch learning: several training samples are fed in at once and the resulting outputs are evaluated together. So we also want a version of cross-entropy that can be computed over the results from multiple training samples at the same time.

Consider a set of multiple training samples like the following.

Since there are multiple training samples, we introduce a Sample No. column so that each sample can be told apart. For readability, the correct answer in each row is highlighted with a background color.

Feeding these four samples into the model one by one and computing the outputs gives the following. A Predicted column has been added on the right.

Now let's compute the cross-entropy for these four samples.

If we compute the cross-entropy for each sample individually and add the values up, we get the total cross-entropy over the four samples. There is nothing difficult about it, so we will simply compute the cross-entropy one sample at a time and sum them.

First, let's do it the straightforward way.

Computing the cross-entropy for sample 1

The cross-entropy for the first sample is as follows. To make the sample number explicit, we write the cross-entropy \(E\) as \(E_{1}\).

$$
\begin{aligned}
\ E_{1} = &- \sum_{k=1}^{K} t_{k} \log y_{k} &\
&= - ( t_{1} \log y_{1} + t_{2} \log y_{2} + t_{3} \log y_{3}) & \
&= - ( 1 \cdot \log 0.33 + 0 \cdot \log 0.32 + 0 \cdot \log 0.35) \
&= \log 0.33 = -0.481486 \
\
&K: \text{number of classes},\quad t_{k}: \text{ground-truth label},\quad y_{k}: \text{predicted value computed by the model}&
\end{aligned}
$$

Computing the cross-entropy for sample 2

Similarly, computing \(E_{2}\) gives

$$
\begin{aligned}
\ E_{2} = &- \sum_{k=1}^{K} t_{k} \log y_{k} &\
&= - ( t_{1} \log y_{1} + t_{2} \log y_{2} + t_{3} \log y_{3}) & \
&= - ( 0 \cdot \log 0.30 + 1 \cdot \log 0.36 + 0 \cdot \log 0.34) \
&= \log 0.36 = -0.443697 \
\end{aligned}
$$

Computing the cross-entropy for sample 3

Similarly, computing \(E_{3}\) gives

$$
\begin{aligned}
\ E_{3} = &- \sum_{k=1}^{K} t_{k} \log y_{k} &\
&= - ( t_{1} \log y_{1} + t_{2} \log y_{2} + t_{3} \log y_{3}) & \
&= - ( 0 \cdot \log 0.37 + 0 \cdot \log 0.31 + 1 \cdot \log 0.32) \
&= \log 0.32 = -0.494850 \
\end{aligned}
$$

Computing the cross-entropy for sample 4

Similarly, computing \(E_{4}\) gives

$$
\begin{aligned}
\ E_{4} = &- \sum_{k=1}^{K} t_{k} \log y_{k} &\
&= - ( t_{1} \log y_{1} + t_{2} \log y_{2} + t_{3} \log y_{3}) & \
&= - ( 0 \cdot \log 0.34 + 1 \cdot \log 0.33 + 0 \cdot \log 0.33) \
&= \log 0.34 = -0.46852 \
\end{aligned}
$$

Let's plug in concrete values. We will look at the sample with sample number \(i=1\) from earlier.

$$
\begin{aligned}
\boldsymbol{E_{1}} = &- \boldsymbol{t} \cdot \log (\boldsymbol{y})& \
=& -
\begin{pmatrix}
t_{1} \ t_{2} \ t_{3}
\end{pmatrix}
\log
\begin{pmatrix}
y_{1} \ y_{2} \ y_{3}
\end{pmatrix}
&
\\
=& -
\begin{pmatrix}
1 \ 0 \ 0
\end{pmatrix}
\log
\begin{pmatrix}
0.33 \ 0.32 \ 0.35
\end{pmatrix}
&
\\
=& -
\begin{pmatrix}
\log(0.33) \ 0 \ 0
\end{pmatrix}
&
\\
=&
\begin{pmatrix}
-0.481486 \ 0 \ 0
\end{pmatrix}
&
\end{aligned}
$$

Now, the sum of the four individually computed cross-entropies \(E_{1}\), \(E_{2}\), \(E_{3}\), and \(E_{4}\) is the total cross-entropy over the four samples. Writing this as \(E_{sum}\), we have

$$
E_{sum} = E_{1} + E_{2} +E_{3} +E_{4}
$$

Expressed with \(\sum\), using \(i\) as the sample number, this becomes

$$
E_{sum} = \sum_{i=1}^4 E_{i}
$$

We had four samples here, but generalizing to \(N\) samples gives

$$
E_{sum} = \sum_{i=1}^N E_{i} \tag{6.1}
$$

By the way, \(t_{k}\) and \(y_{k}\) originally carried the class number as their subscript. In this example we wanted to classify into three classes, "dog", "fox", and "wolf", so we used \(k={1},k=2,k=3\).
That was fine for a single sample, but now that we have four samples, we add the sample number \(i\) as an additional subscript so that each \(t_{k}\) and \(y_{k}\) can be uniquely identified.

Concretely, we extend the notation as \(t_{k}\) → \(t_{ik}\) and \(y_{k}\) → \(y_{ik}\), as shown below.

With this notation,

• \(t_{ik}\) is the \(k\)-th element of the \(i\)-th training sample
• \(y_{ik}\) is the \(k\)-th element of the model's output (predicted value) when the \(i\)-th training sample is fed in

Therefore, the cross-entropy for the \(i\)-th sample is Equation \((5.2)\) with the subscript \(i\) added, as follows.

$$
E_{i} = - \sum_{k=1}^{K} t_{ik} \log y_{ik} \tag{6.2}
$$

From \(\text{Equation }(6.1)\),

$$
\begin{aligned}
E_{sum} =& \sum_{i=1}^N E_{i} &\
\end{aligned}
$$

so the total cross-entropy summed over N samples is

$$
\begin{aligned}
E_{sum} = & - \sum_{i=1}^N \sum_{k=1}^{K} t_{ik} \log y_{ik} &\
\end{aligned}
$$

The expression above is a total over \(N\) samples. To make it comparable across different sample counts, we divide by N to take the mean of the cross-entropies \(E_{i}\), and the batch version, i.e. the multi-sample version of the cross-entropy function \(E\) is defined as follows.

$$
E = - \frac{1}{N} \sum_{i=1}^N \sum_{k=1}^{K} t_{ik} \log y_{ik} \tag{6.3}
$$

$$
\begin{aligned}
\
& N: \text{number of samples}& \
&i: \text{sample number}& \
&K: \text{number of classes}& \
&k: \text{class number}& \
&t_{ik}: \text{ground-truth label (teacher signal) of the } k\text{-th class for the } i\text{-th sample}& \
&y_{ik}: k\text{-th predicted value in the output for the } i\text{-th input sample}& \
\end{aligned}
$$

At last we have defined the cross-entropy function for multi-class classification introduced at the beginning. This is known as Categorical Cross-Entropy.

How was this installment?

We successfully derived the cross-entropy for multi-class classification with N samples.
Next time, we will derive the cross-entropy for binary classification.

References
https://journal.qualiteg.com/books/

navigation