# What's an entropy measure for the sequence $0, 1, 0, 1, 0, 1, …$?

I'm puzzled by the measure $$E(S) = -\sum p(i) \lg p(i)$$ because I'm considering a source emitting $$0, 1, 0, 1, 0, ...$$ and having the maximum entropy as given by $$E(S)$$. Shouldn't $$E(S)$$ measure the quantity of information? The sequence that just repeats $$1$$ after $$0$$ seems to me to convey very little information --- contained in this very phrase.

Where am I wrong in my understanding of entropy and of $$E(S)$$?

• What is the connection to cryptography? This appears to be a question that is better suited for the computer science stackexchange. – Ella Rose Feb 5 '19 at 1:40
• there are different definitions for entropy, in this case, you can rewrite that is 0,1,repeat, so, for cryptographic definitions of entropy, 2 bytes maybe? – Richie Frame Feb 5 '19 at 1:51
• Entropy (in the sense of the definition you give) is a property of probability distributions; it makes no sense to speak of the entropy of a sequence. – fkraiem Feb 5 '19 at 2:00
• @yyyyyyy Why can it not be measures objectively? For example the size of the smallest algorithm (including data) on a given Turing Complete machine that can output the original data? (that would be a few bytes in this case) – RocketNuts Feb 5 '19 at 4:38
• You are probably asking about the complexity of that sequence. The sequence itself has no entropy, and wouldn't even if it were perfectly random. The generation method, however, can exhibit entropy and that entropy is measured as the binary logarithm of the number of possible states the algorithm can be in. – forest Feb 5 '19 at 5:58

\begin{align*} E(S)&=-p_0\log p_0 - p_1\log p_1\\ &=-p_0\log p_0 - (1-p_0)\log (1-p_0)\\ &\mathrel{\mathop{=}_{p_0\to1}} -1\log 1 - 0\\ &=-1\cdot0=0 \end{align*}
• Interesting. I don't seem to agree. What's the probability of getting a zero given no other information? I say it's $1/2$. What's the probability of getting a zero given a one was just produced by the source? I say it's $1$. The formula $E(S)$ uses $p_i$ where $p_i$ is the proportion of symbol $i$ in the source. What do you say? – user45491 Feb 5 '19 at 11:20
• @user45491 Welcome to entropy. It's pretty much an open question how to calculate it for a complex sequence. The Shannon log formula is pretty difficult to use as determining $i$ is well neigh impossible. That's why you're finding that $p("1") = \frac{1}{2}$, yet $p("01") = 1$ and that all the answers are different. – Paul Uszak Feb 5 '19 at 16:38