CE vs. K-CE · Joseph S. Miller

Abstract

Ordinary cross-entropy tries to fit every distinction in a probability law with infinite resolution. That is sometimes what is wanted, but when a model has only a few degrees of freedom, CE may use them to resolve distinctions that the downstream task doesn't actually value instead of fitting more useful things. We can use a similarity kernel to change the relative cost of those errors.

Setting

The previous feature-engineering essay is about transformations that preserve the distinctions in a task kernel. Here I will instead discuss fitting a model given a task kernel (e.g. in the output coordinates).

For a model $Q$, the kernel-smoothed mass around $z$ is its typicality

(KQ)(z):=\int K(z,z')\,Q(dz').

Ordinary cross-entropy fits the complete law with an implicit infinitely narrow identity kernel. When the corresponding entropy is concave, the scoring-rule essay derives the proper tangent score $S_K$ that also uses the kernel. Averaging over the true distribution, I will write its expected risk as

\mathcal C_K(P,Q):=\mathbb E_{Z\sim P}S_K(Q,Z),

and call it $K$-cross-entropy, or $K$-CE. The score combines negative log typicality $-\log(KQ)(Z)$ with the correction that makes it proper. With an identity kernel on a discrete space, $K$-CE is ordinary cross-entropy. On a continuous space, the fitted models approach the ordinary CE fit with the appropriately narrow kernel.

How CE and K-CE differ

If the model family $\mathcal Q$ contains $P$, both objectives agree. Ordinary CE is minimized at $P$, and proper $K$-CE is also minimized at $P$. The objectives can disagree when the model is restricted: perhaps the model has a fixed number of components, the model is regularized, or the data only support a small model. In that case it has to trade off against different errors. CE will trade between those errors at the exact local density resolution; $K$-CE discounts errors between similar outcomes.

Write ordinary cross-entropy as $\mathcal C_{\log}$. The models in this family that best fit the population under each score are

\begin{aligned} Q_{\mathrm{CE}} &\in\arg\min_{Q\in\mathcal Q}\mathcal C_{\log}(P,Q),\\ Q_K &\in\arg\min_{Q\in\mathcal Q}\mathcal C_K(P,Q). \end{aligned}

I'll compare these two models under both scores:

\begin{aligned} \Delta_K &:=\mathcal C_K(P,Q_{\mathrm{CE}})-\mathcal C_K(P,Q_K),\\ \Delta_{\log} &:=\mathcal C_{\log}(P,Q_K)-\mathcal C_{\log}(P,Q_{\mathrm{CE}}). \end{aligned}

$\Delta_K$ is how much $K$-CE falls when we fit using $K$-CE instead of CE. $\Delta_{\log}$ is how much CE falls when we fit using CE instead of $K$-CE.

Gaussian Mixture Examples

In contrast to the finite-sample Gaussian-mixture singularity in the scoring-rule essay which demonstrates the problems of infinite costs and rewards when fitting to sample data, the focus here is on the shape of the tradeoffs between different models to population distributions. Consider again a one-dimensional population with four narrow Gaussian modes:

P=0.38\,\mathcal N(-1,0.08^2) +0.32\,\mathcal N(-0.3,0.08^2) +0.18\,\mathcal N(1,0.08^2) +0.12\,\mathcal N(2.3,0.08^2).

The first two modes contain $70\%$ of the population and are relatively close. The two modes to their right are farther apart for this task. We'll use a Laplace kernel, one of the concave cases covered by the scoring rule,

K_\ell(z,z')=\exp\!\left(-\frac{|z-z'|}{\ell}\right), \qquad \ell=0.6.

We now fit $Q$ directly to the true density $P$ with only three Gaussian components, where each can have its own weight, mean, and width.

CE uses two components to reproduce the large nearby modes almost exactly, and uses its third component to cover both right-hand modes with one wide Gaussian centered near $1.50$. Broad-kernel $K$-CE makes the opposite tradeoff, using one wide component centered near $-0.68$ for the nearby pair.

A four-mode true density and two three-component fits. CE resolves the two large nearby modes and merges the two modes on the right. K-CE merges the nearby pair and resolves the two modes on the right. A second panel shows their task typicality fields.

Smoothing at the task scale reduces the importance of the low density region between the left two modes, so $K$-CE prioritizes fitting the more distinct regions to the right.

Kernel and model family	$\Delta_K$ (bits)	$\Delta_{\log}$ (bits)
Narrow ($\ell=0.08$), 3 components	$<0.001$	$<0.001$
Broad ($\ell=0.6$), 3 components	$0.021$	$0.231$
Broad ($\ell=0.6$), 4 components	$0.000$	$0.000$

With a narrow kernel, $K$-CE and CE place the three components in nearly the same locations, indicating that the more computationally simple CE is sufficient. With four components, the truth is in the model family and both objectives recover it exactly, so fitting with local CE is also preferred. Different models are chosen when the kernel gives meaningful partial credit and the capacity is too small to fit everything.

CE vs. $K$-CE

For models that are expensive to fit, it seems reasonable to fit candidate models or model families by ordinary CE and compare their predictions with held-out $K$-CE when transported appropriately. This is because $\mathcal C_K$ gives us a common scale for comparing models on the task, even when their coordinates and ordinary cross-entropies differ. Evaluating the kernel score may still be expensive, but it avoids optimizing every model through the nonlocal objective.

The CE fit can also be a good starting point for direct $K$-CE fitting. If a promising model still has some task regret, one could initialize at the CE minimizer (equivalently, the likelihood maximizer) and fine-tune with $K$-CE. That way, we can try the more expensive fit on promising models and see whether fine-tuning to the task is worth the extra computation.