In the history of probability theory there has been considerable thought devoted to describing the distribution for infinite sums of random variables. In this effort two classic approaches have been developed: (1) Local Limit Theorems and (2) Central Limit Theorems.

Each of these approaches contain many theorems. All of these theorems, in the classical understanding, prove, under various assumptions, that the limiting distribution of sums of random variables is normal. (It's worth noting that more recent work has applied central limit techniques to prove limiting distributions other than normal, but these are not the classical results). To begin to understand these theorems we will first state two theorems as presented in [1].

Theorems

The Local Demoivre-Laplace Theorem

Given an i.i.d. sequence of Bernoulli random variables \( \xi_1, \xi_2, \ldots,\xi_n \) with \( p \) probability of success \[ \lim_{n\to\infty} \mathbf{P}\left(\sum_{i=1}^n\xi_i=k\right) = \frac{1}{\sqrt{2\pi np(1-p)}} e^{\left(-x^2/2\right)}, \] when \( x = \frac{k-np}{\sqrt{np(1-p)}} \) and it is bound to a finite interval.

Lindeberg's Central Limit Theorem

Given a sequence of mutually independent random variables \(\xi_1, \xi_2, \ldots,\xi_n\) with finite \(\mathbf{E}\xi\) and \(\mathbf{Var}(\xi)\) \[\lim_{n\to\infty} \mathbf{P}\left(\frac{1}{\sigma}\sum_{i=1}^n(\xi_i -\mathbf{E}\xi_i) < k \right) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{k} e^{\left(-x^2/2\right)}dx, \] when \( \lim_{n\to\infty}\frac{1}{\sigma^2}\sum_{i=1}^n\mathbf{E}\left[(\xi_i - \mathbf{E}\xi_i)^2 \mathbf{1}_{\vert\xi_i-\mathbf{E}\xi_i\vert > \tau\sigma} \right] = 0\), \(\sigma^2=\mathbf{Var}(\sum_{i=1}^n\xi_i) \) and \(\tau > 0\).

Analysis

Comparing The Two Approaches

We can see above that the difference in the two approaches is simply how the distribution is centered and normalized. In the local theorem \(k\) is centered and normed, and the random variables are left unchanged. In the central limit theorem the random variables are centered and normed and \(k\) is left unchanged. This difference is simply a techical matter with very little effect on the implications of the theorems. For this reason both approaches are often discussed as if they are both part of the same school of thought.

Understanding The Lindeberg Condition

In Lindeberg's Central Limit Theorem above there is an important condition that deserves special attention, \[ \lim_{n\to\infty}\frac{1}{\sigma^2}\sum_{i=1}^n\mathbf{E}\left[(\xi_i - \mathbf{E}\xi_i)^2 \mathbf{1}_{\vert\xi_i-\mathbf{E}\xi_i\vert > \tau\sigma} \right] = 0\ \text{for}\ \tau > 0. \]

This condition is known as Lindeberg's Condition, and it has been shown to be the sufficient condition for the sum of a sequence of mutually independent random variables with finite mean and variance to converge to a normal distribution. To explain this condition [1] proves \[ \mathbf{P}\left(\max_{1 \le i \le n} \vert \xi_i - \mathbf{E}\xi_i \vert \ge \tau\sigma \right) \le \frac{1}{\tau^2\sigma^2}\sum_{i=1}^n\mathbf{E}\left[(\xi_i - \mathbf{E}\xi_i)^2 \mathbf{1}_{\vert\xi_i-\mathbf{E}\xi_i\vert > \tau\sigma} \right]\ \text{for}\ \tau > 0. \]

For any sequence of random variables satisfying the Lindeberg Condition the right side of this inequality will go to 0 as \(n \to \infty\), implying that the sequence will be contained within the interval \((-\tau\sigma, \tau\sigma)\) with probability 1 as \(n \to \infty\). That is, that no single random variable will dominate in the limit.

Remembering Finite Mean and Variance

Regarding central limit theorems it should be emphasized that they can only be applied to random variables with well-defined, finite means and variances. Otherwise it would not be possible to center and norm each random variable as the theorems assume can be done. For example, central limit theorems cannot be applied to a sequence of Cauchy random variables (Cauchy random variables do not have a well-defined mean or variance). In fact, it can be shown that the limiting distribution for the sum of a sequence of Cauchy random variables is itself a Cauchy random variable rather than a normal distribution.

Estimating A Random Variable's Mean

One important application of central limit theorems is estimating the mean of a random variable from i.i.d. samples. To show this we require the following claim regarding normal distributions: given a random variable \(X \sim{} \mathcal{N}(\mu, \sigma^2) \) we also have \( aX \sim{} \mathcal{N}(a\mu, a^2\sigma^2)\ \forall a \in \mathbb{R} \).

Now, consider an i.i.d. sequence of random variable \(\xi_1, \xi_2, \ldots, \xi_n \) satisfying the Lindeberg Condition and with finite \(\mathbf{E}\xi = \mu\) and \(\mathbf{Var}(\xi) = \sigma^2\). By the Lindeberg Central Limit Theorem we have \(\lim_{n\to\infty}\sum_{i=1}^n \xi_i = \mathcal{N}(n\mu,n\sigma^2)\). Combining this result with the above claim about normal distributions we can get \[\lim_{n\to\infty}\sum_{i=1}^n \frac{1}{n}\xi_i = \mathcal{N}(\mu,\sigma^2/n).\]

References

[1] Gnedenko, Boris V. Theory of probability. Routledge, 2018.