David's raw ML reference notes

Gundersen 2020

4 min read

Summary

An insightful blog post that connects the idea of a moment in statistics to physical ideal like ‘momentum’ and ‘moment of inertia’. Starts with a formal definition and an etymology; progresses to some additional terminology; then gives an account of the first five moments:

  1. 0th moment of a function (total mass)
  2. 1st moment of a function (mean)
  3. 2nd moment of a function (variance)
  4. 3rd moment of a function (skewness)
  5. 4th moment of a function (kurtosis)

It concludes with a discussion of higher-order moments and Moment-generating functions.

Metadata

  • Author: Gregory Gundersen
  • Full Title: Understanding Moments
  • Category:articles
  • Summary: The text discusses moments in statistics, including mean, variance, skewness, and kurtosis, as ways to describe the shape of a distribution. Moments quantify a distribution’s location, scale, and shape, with skewness measuring tail size and kurtosis capturing tail weight. Higher moments recapitulate information from lower moments, with skewness and kurtosis commonly used to describe distributions.
  • URL: https://readwise.io/reader/document_raw_content/163420825

Highlights

  • According to Wiktionary, the words “moment” and “momentum” are doublets or etymological twins (View Highlight)
  • Both come from the Latin word “movimentum,” meaning to move, set in motion, or change (View Highlight)
  • In physics, the moment of inertia is a measure of rotational inertia or how difficult it is to change the rotational velocity of an object (View Highlight)
  • moments quantify three parameters of distributions: location, shape, and scale. (View Highlight)
  • the first moment describes a distribution’s location, the second moment describes its scale, and all higher moments describe its shape. (View Highlight)
  • a raw moment is a moment about the origin ( (2) c = 0), and a central moment is a moment about the distribution’s mean (c = E[X]) (View Highlight)
  • standardized moment (View Highlight)
  • . Standardization makes the moment both location- and scale-invariant. (View Highlight)
  • given a statistical model, parameters summerize data for an entire population, while statistics summarize data from a sample of the population (View Highlight)
  • , the zeroth moment captures the fact that probability distributions are normalized quantities, that they always sum to one regardless of their location, scale, or shape (View Highlight)

New highlights added April 16, 2024 at 6:09 PM

  • Another way to think about the first moment is that it is that it is the center of mass of a probability distribution. (View Highlight)
  • the first moment tells us how far away from the origin the center of mass is. (View Highlight)
  • variance (View Highlight)
    • Note: Second central moment
  • any data point less than a standard deviation from the mean (i.e. data near the center) results in a standard score less than ; this is then raised to the third power, making the absolute value of the cubed standard score even smaller. In other words, data points less than a standard deviation from the mean contribute very little to the final calculation of skewness. Since the cubic function preserves sign, if both tails are balanced, the skewness is zero. Otherwise, the skewness is positive for longer right tails and negative for longer left tails. 1 (View Highlight)
  • Standardizing the moment in (15) is important because skewness is both location- and scale-invarant. In other words, two distributions can have the same mean and variance but different skewnesses. (View Highlight)
  • , skewness is invariant to sign-preserving scaling transformations. (View Highlight)
  • kurtosis is a measure of the combined size of the tails relative to whole distribution. (View Highlight)
  • the typical metric is the fourth standardized moment, (View Highlight)
  • long tails on either side dominate the calculation. (View Highlight)
  • kurtosis measures tailedness, not peakedness. (View Highlight)
  • excess kurtosis (View Highlight)
  • Odd-powered standardized moments quantity relative tailedness and even-powered standardized moments quantify total tailedness. (View Highlight)
  • a random variable’s moment-generating function is an alternative specification of its distribution (View Highlight)
  • moment-generating function (View Highlight)
  • , the th derivative evaluated at the origin is the th moment. (View Highlight)
  • We can compute moments from the MGF by taking derivatives. (View Highlight)

Graph View

Backlinks