Mathematical Proofs

Proof 1: Derivative of $\ln p$

The derivative of $\ln p$ is defined as:

\frac{d}{d p} \ln p = lim_{Δ p \to 0} \frac{\ln (p + Δ p) - \ln p}{Δ p}

Using the logarithm difference rule:

\ln (p + Δ p) - \ln p = \ln (\frac{p + Δ p}{p})

Rewriting the derivative:

\frac{d}{d p} \ln p = lim_{Δ p \to 0} \frac{\ln (1 + \frac{Δ p}{p})}{Δ p}

Using the limit property:

lim_{x \to 0} \frac{\ln (1 + x)}{x} = 1

where we set $x = \frac{Δ p}{p}$ , we get:

\ln (1 + \frac{Δ p}{p}) \approx \frac{Δ p}{p}

Thus, our derivative simplifies to:

\frac{d}{d p} \ln p = \frac{1}{p}

H (p_{1}, p_{2}, \dots, p_{n}) = - \sum_{i = 1}^{n} p_{i} \ln p_{i}

with the constraint:

\sum_{i = 1}^{n} p_{i} = 1

L (p_{1}, p_{2}, \dots, p_{n}, λ) = - \sum_{i = 1}^{n} p_{i} \ln p_{i} + λ (\sum_{i = 1}^{n} p_{i} - 1)

\frac{\partial}{\partial p_{i}} (- p_{i} \ln p_{i}) = - (\ln p_{i} + 1)

\frac{\partial}{\partial p_{i}} λ \sum_{i = 1}^{n} p_{i} = λ

- (\ln p_{i} + 1) + λ = 0

\ln p_{i} = λ - 1

p_{i} = e^{λ - 1}

Since $\sum p_{i} = 1$ , we solve:

n e^{λ - 1} = 1

e^{λ - 1} = \frac{1}{n}

p_{i} = \frac{1}{n}, \forall i

This confirms entropy is maximized when all probabilities are equal.

Let:

p_{i} = \frac{1}{n} + δ_{i}, where \sum_{i = 1}^{n} δ_{i} = 0

Expanding entropy using a second-order Taylor series:

H (p + δ) = H (p^{*}) + \sum_{i} \frac{\partial H}{\partial p_{i}} |_{p^{*}} (p_{i} - p_{i}^{*}) + \frac{1}{2} \sum_{i, j} \frac{\partial^{2} H}{\partial p_{i} \partial p_{j}} |_{p^{*}} (p_{i} - p_{i}^{*}) (p_{j} - p_{j}^{*}) + O (δ^{3})

\frac{\partial H}{\partial p_{i}} = - (\ln p_{i} + 1)

At $p_{i}^{*} = 1 / n$ :

\frac{\partial H}{\partial p_{i}} |_{p^{*}} = \ln n - 1

Since $\sum (p_{i} - p_{i}^{*}) = 0$ , this term vanishes.

\frac{\partial^{2} H}{\partial p_{i}^{2}} = - \frac{1}{p_{i}}

At $p_{i}^{*} = 1 / n$ :

\frac{\partial^{2} H}{\partial p_{i}^{2}} = - n

So the second-order term is:

\frac{1}{2} \sum_{i} (- n) (p_{i} - p_{i}^{*})^{2} = - \frac{n}{2} \sum_{i} (p_{i} - p_{i}^{*})^{2}

Since $- \sum_{i} (p_{i} - p_{i}^{*})^{2} \geq 0$ , this term is always negative, confirming concavity.

Thus, entropy is maximized at $p_{i} = 1 / n$