lecture05: Entropy

Topics covered

• Entropy
• Entropy Example 1: Bernoulli RV
• Entropy Example 2: \cite[p.14]{CoverThomas}
• Entropy Example 3: \url{https://xkcd.com/936/}
• Joint Entropy
• Conditional Entropy
• Conditional Entropy: examples
• Relative Entropy
• Relative Entropy Example: \cite[p.17]{CoverThomas}
• More Complex Entropy Example: \cite[p.17]{CoverThomas}
• Entropy
• Context
• Information
• Shannon and Information Theory
• Later developments
• Today