Q: What's the relationship between cross entropy, entropy and self-information

In Information Theory, these three concepts form a hierarchy. You can think of them as building blocks: Self-Information is the atom, Entropy is the molecule built from those atoms, and Cross-Entropy is how that molecule interacts with a different molecule.

Here is the relationship in a nutshell, followed by the deep dive: $$\text{Cross-Entropy} = \text{Entropy} + \text{KL Divergence}$$

The Atom: Self-Information

Before we can understand the average, we must understand the individual unit. Self-information (or “surprisal”) measures the surprise associated with a single outcome.

最近工作上要用到C++。在学校的时候对C++的基本用法已经有了了解，调研了一下又哪些比较好的书，分享给大家。

A Tour of C++

这本书只有短短的192页，但是把很多的C++新特性都囊括在内。适合之前有编程经验的用户。