Mathematical Proofs

Proof 1: Derivative of \( \ln p \)

The derivative of \( \ln p \) is defined as:

\[ \frac{d}{dp} \ln p = \lim_{\Delta p \to 0} \frac{\ln(p + \Delta p) - \ln p}{\Delta p} \]

Using the logarithm difference rule:

\[ \ln(p + \Delta p) - \ln p = \ln \left( \frac{p + \Delta p}{p} \right) \]

Rewriting the derivative:

\[ \frac{d}{dp} \ln p = \lim_{\Delta p \to 0} \frac{\ln(1 + \frac{\Delta p}{p})}{\Delta p} \]

Using the limit property:

\[ \lim_{x \to 0} \frac{\ln(1 + x)}{x} = 1 \]

where we set \( x = \frac{\Delta p}{p} \), we get:

\[ \ln(1 + \frac{\Delta p}{p}) \approx \frac{\Delta p}{p} \]

Thus, our derivative simplifies to:

\[ \frac{d}{dp} \ln p = \frac{1}{p} \]

Proof 2: Find Maximizing Entropy Using Lagrange Multipliers using Proof 1

Step 1: Define the Entropy Function

\[ H(p_1, p_2, \dots, p_n) = - \sum_{i=1}^{n} p_i \ln p_i \]

with the constraint:

\[ \sum_{i=1}^{n} p_i = 1 \]

Step 2: Construct the Lagrange Function

\[ \mathcal{L}(p_1, p_2, \dots, p_n, \lambda) = - \sum_{i=1}^{n} p_i \ln p_i + \lambda \left( \sum_{i=1}^{n} p_i - 1 \right) \]

Step 3: Differentiate the Entropy Term

\[ \frac{\partial}{\partial p_i} (- p_i \ln p_i) = - (\ln p_i + 1) \]

Step 4: Differentiate the Lagrange Constraint

\[ \frac{\partial}{\partial p_i} \lambda \sum_{i=1}^{n} p_i = \lambda \]

Step 5: Solve for \( p_i \)

\[ - (\ln p_i + 1) + \lambda = 0 \] \[ \ln p_i = \lambda - 1 \] \[ p_i = e^{\lambda - 1} \]

Since \( \sum p_i = 1 \), we solve:

\[ n e^{\lambda - 1} = 1 \] \[ e^{\lambda - 1} = \frac{1}{n} \] \[ p_i = \frac{1}{n}, \quad \forall i \]

This confirms entropy is maximized when all probabilities are equal.

Confirming Maximum Entropy Using Taylor Series using Proof 1 and 2

Step 1: Define Small Deviations

Let:

\[ p_i = \frac{1}{n} + \delta_i, \quad \text{where} \quad \sum_{i=1}^{n} \delta_i = 0 \]

Step 2: Taylor Expansion of \( H(p) \)

Expanding entropy using a second-order Taylor series:

\[ H(p + \delta) = H(p^*) + \sum_{i} \frac{\partial H}{\partial p_i} \Big|_{p^*} (p_i - p_i^*) + \frac{1}{2} \sum_{i,j} \frac{\partial^2 H}{\partial p_i \partial p_j} \Big|_{p^*} (p_i - p_i^*) (p_j - p_j^*) + O(\delta^3) \]

Step 3: Compute the First-Order Term

\[ \frac{\partial H}{\partial p_i} = - (\ln p_i + 1) \]

At \( p_i^* = 1/n \):

\[ \frac{\partial H}{\partial p_i} \Big|_{p^*} = \ln n - 1 \]

Since \( \sum (p_i - p_i^*) = 0 \), this term vanishes.

Step 4: Compute the Second-Order Term

\[ \frac{\partial^2 H}{\partial p_i^2} = -\frac{1}{p_i} \]

At \( p_i^* = 1/n \):

\[ \frac{\partial^2 H}{\partial p_i^2} = -n \]

So the second-order term is:

\[ \frac{1}{2} \sum_i (-n) (p_i - p_i^*)^2 = -\frac{n}{2} \sum_i (p_i - p_i^*)^2 \]

Since \( -\sum_i (p_i - p_i^*)^2 \geq 0 \), this term is always negative, confirming concavity.

Thus, entropy is maximized at \( p_i = 1/n \)