F1 - maximum entropy principle

considering a random variable, $X$ , with known states, $x_{n}$ , but unknown probabilities, $p_{n}$
also, $⟨ X ⟩$ is known
in the case of $X \in {1, 2, 3}$ , $⟨ X ⟩ = \bar{x} = 2$
there can be multiple possible values for the probabilities

maximum entropy principle

the best representing probability distribution is the one that maximizes the entroppy

the entropy:

H (p_{i}) = - \sum_{i} p_{i} \log p_{i}

using the method of lagrange multipliers:

\begin{aligned} d [- p_{i} \log p_{i} - λ (\sum_{i = 1}^{3} i p_{i} - \bar{x}) - μ (\sum_{i = 1}^{3} p_{i} - 1)] & = 0 \\ \sum_{i = 1}^{3} (- \log p_{i} - 1 - λ i - μ) d p_{i} & = 0 \end{aligned}

the term in the curly brackets must be zero

p_{i} = \exp (- μ - 1 - λ_{i})

using the constrains, it can be found that:

\begin{matrix} p_{i} = \frac{\exp (λ (1 - i))}{1 + e^{- λ)} + e^{- 2 λ}} \\ \bar{x} = \frac{1 + 2 e^{- λ} + 3 e^{- 2 λ}}{1 + e^{- λ} + e^{- 2 λ}} \end{matrix}

therefore, the probabilities take the values:
- $(1, 0, 0)$ when $\bar{x} = 1$
- $(0, 0, 1)$ when $\bar{x} = 3$
- $(1 / 3, 1 / 3, 1 / 3)$ when $\bar{x} = 2$

generalized

considering $X \in {x_{1}, \dots, x_{n}}$ with unknown probabilities $p_{i}, \dots, p_{n}$
also, there are $m$ constraint functions: $⟨ f_{k} (X) ⟩ = F_{k}$ with $1 \leq k \leq m < n$
using the method of lagrange multipliers:

p_{i} = \exp (- μ + 1 - \sum_{k = 1}^{m} λ_{k} f_{k} (x_{i})) = e^{1 - μ} \exp (- \sum_{k = 1}^{m} λ_{k} f_{k} (x_{i}))

the sum of which has to be $1$
the partition function is defined as:

Z = \sum_{i = 1}^{n} \exp (- \sum_{k = 1}^{m} λ_{k} f_{k} (x_{i}))

hence:

\exp (1 - μ) = \frac{1}{Z} ⟹ 1 - μ = \log Z

the other constraints:

F_{k} = - \frac{1}{Z} \frac{\partial Z}{\partial λ_{k}} = - \frac{\partial \log Z}{\partial λ_{k}} $ $ $ $ ∴ p_{i} = \frac{1}{Z} \exp (- \sum_{k = 1}^{m} λ_{k} f_{k} (x_{i}))

the entropy is now a function of the constraint values, $F_{k}$ , and can be rewritten as $S (F_{1}, \dots, F_{m})$