Distribution notation in probability and information theory

Random variable

: A random variable, which can be discrete or continuous.

Probability mass function (pmf)

: The probability mass function of a discrete random variable . It gives the probability that takes the value .

Probability density function (pdf)

: The probability density function of a continuous random variable . It describes the likelihood of taking a specific value, such that

Cumulative distribution function (cdf)

: The cumulative distribution function of . It represents the probability that takes a value less than or equal to .

Shannon entropy

: The Shannon entropy of the random variable , defined for a discrete random variable as

Perplexity

Perplexity is a measure derived from the entropy of a probability distribution. It is defined as the exponentiation of the entropy:

In the context of a discrete random variable with probability mass function , perplexity can be interpreted as the effective number of choices (or the “average branching factor”) represented by the distribution.