A random string and a sonnet can carry the same information? Then what is Shannon measuring?

Accepted answer

1. Vikram Sethi 4 weeks ago

   ✓ Accepted answer Fellow

   Serkan, think of it as a measure of surprise, not of meaning.

   Suppose I tell you the result of a coin toss. There were two equally likely possibilities; "heads" removes that uncertainty. Shannon calls that one bit. The result of a die roll removes more uncertainty, since there were six possibilities, so it carries more bits. The amount depends only on how many possibilities there were and how likely each was, never on what the message is about.

   Now the sonnet. If you treat both strings as sequences of letters where any letter could come next with equal chance, they are equally improbable, so they carry the same information. Interestingly, if you use a model of English, the sonnet is more predictable (after "q" comes "u", and so on), so it carries less Shannon information than the random string. Either way, meaning never enters the calculation.

   Shannon was clear that this was deliberate. Early in his 1948 paper he says that the semantic aspects of communication "are irrelevant to the engineering problem". His problem was how to send messages reliably down a wire. For that, you don't need to know what they mean, any more than the post office needs to read your letters.

   The confusion came later, when the technical word was borrowed as if it carried the everyday sense.

   1. SA Serkan Aydoğdu 4 weeks ago

      Fellow

      Vikram, the post office comparison is what I needed. So the size of the envelope, not the letter. And a love letter and a tax demand can weigh the same.