Shannon's Entropy

This MOOC teaches the math behind Shannon's entropy

This miniMOOC teaches the math behind Shannon's entropy. It was created by Dr. Rivki Gadot (Open University , Lev Academic Center) & Dvir Lanzberg (the lecturer). It was made for undergraduate students, but we believe that anyone with a basic knowledge in calculus and algebra can get over the math in this miniMOOC.Shannon started by asking what is "information" and how it can be measured. Instead of giving a definition, he claimed that any function that measures information must have three properties, and proved that there is a single function that meets these three properties. This function become to be known as, Shannon's entropy.There are six short clips in this miniMOOC. Each clip is accompanied by exercises that let you deepen your understanding. The topics of the six clips are:

  1. The problem - states what was the problem that Shannon tried to solve.
  2. The solution - introduces Shannon's entropy.
  3. The three properties - explains what are the three properties that every function for the amount of information, must have.
  4. Equal probabilities - proves that Shannon's entropy is the only function that has the three properties, if the events' probabilities were equal.
  5. Rational probabilities - proves that Shannon's entropy is the only function that has the three properties, if the events' probabilities were rational numbers.
  6. Real probabilities - proves that Shannon's entropy is the only function that has the three properties, if the events' probabilities were real numbers.

The clips' presentation in PPTX format, and in PDF format (including lecturer's note).

1. The Problem

Information and surprise

This clip states what was the problem that Shannon tried to solve.

  1. How much surprise is there in finding out the outcome of a coin toss?
  2. How much surprise is there in finding out which permutation of an array is sorted?
  3. How much surprised can we get by knowing the weather in Tel-Aviv in August?

After watching the clip, try to solve Part 1 - Questions.
Want to check your answers? Download Part 1 - Answers for answers and explanations.


אולי ימצא חן בעיניך

תחרות המתמטיקה

21 ביוני 2018

תחרות המתמטיקה לבתי הספר היסודיים - SKillz Math

קרא עוד

תחרות רחפנים

14 בנובמבר 2018

תחרות הרחפנים לחטיבות העליונות - Skillz Makerz

קרא עוד

תחרות התכנות

21 ביוני 2018

תחרות התכנות לבתי ספר יסודיים וחט"ב - Skillz Coding Junior

קרא עוד