The Chi-Square Statistic

Not only is this statistic useful, but you can impress your associates with your expertise in statistics by just dropping the name


Chi-square, or χ2, is a statistic calculated from the values of a reasonably large sample that can be used to discover whether the number of values in various ranges or bins is unlikely to be consistent with prior assumptions about the distribution of the data. It cannot prove, and no statistic can, that the data obey a certain distribution. A statistical distribution is a property of a population, not of a limited sample, and cannot be determined from a sample alone. The best way to explain χ2 is to begin with a concrete example. It is much easier to do than to explain!

Suppose, therefore, that we have a class of 50 students who have sat a certain examination. We analyze the 50 marks in the usual way, and find that the mean M is 70% and the standard deviation S is 16%. We have arbitrarily decided to award letter grades in bands of a constant width of S/2, so that 90% and up is an A+, 82-89% A, 74-81% B, 66-73% C, 58-66% D, 50-57% D-, 49% and below F. What we want to know is if the number of marks falling into each band is consistent with a normal distribution of marks. I do not suggest that this is any way to grade, or that this is a good way to test for a normal distribution, but it will be a good illustration of χ2, and is quite practical.

The first thing to do is to find out how many marks we would expect to fall into each bin on the assumption of a normal distribution. We start with a standard normal distribution of mean zero and unit standard deviation, for which tables have been calculated. We divide the x-axis into bands of width σ/2 or 0.5, and find the area of the normal distribution φ(x) in each band. The tables give the integral of the normal distribution, Φ(x), which makes the job very easy. The fraction in the C bin is then Φ(0.25) - Φ(-0.25), that in the B bin Φ(0.75) - Φ(0.25), and so on. The tables only give Φ(x) for x > 0.5, since the normal distribution is symmetric. This also means that the number to be expected to fall in the F bin equals that in the A+ bin, the D- bin that in the A bin, and the D bin that in the B bin. The fractions expected are: A+,F: 0.1046; A,D-: 0.1210; B,D: 0.1747; C: 0.1974. These fractions will do for any mean and standard deviation of test marks.

Now we find the expected numbers in each bin by multiplying the number of students by the fractions we have just found. The results are, from A+ to F: 5.23, 6.05, 8.74, 9.87, 8.74, 6.05, 5.23, rounding to the nearest hundredth of a student. The observed values will be integral, but there is no reason the expected values have to be integral. Let us assume that the actual results are: 6, 4, 10, 9, 12, 5, 4. If we call an expected value E and an observed value O, we now form the quantities (O - E)2/E for each bin. The results are: 0.1134, 0.6946, 0.1816, 0.0767, 1.2160, 0.1822, 0.2893. Now we add these all up, and the result is χ2 = 2.754. It is obvious that every departure from our expectations is going to make a positive contribution to χ2, so the larger the value, the more likely that our sample is not consistent with a normal distribution.

But, how large is large? To find out, we must first determine the number of degrees of freedom are involved in χ2. We started with 7 bins, but the numbers in the bins are not all independent. For one thing, they must add up to 50, the total number of students. For another, they must give a mean of 70% and a standard deviation of 16%. Therefore, there are only 7 - 3 or 4 degrees of freedom. This is the only difficult thing in a χ2 analysis, and it is important to get it right. It would be so much more difficult to find the proper 4 independent numbers that there is no reasonable alternative.

Finally, we consult a table of χ2, where the values corresponding to different probabilities and degrees of freedom are set out. The probability is the probability that the value of χ2 due to chance variations does not exceed the tabular value. At a 1% level, the value χ2 = 13.28 is not exceeded for 4 degrees of freedom. Even at a 10% level, the figure is 7.78. This means that in 1 of 10 trials, χ2 would exceed 7.78. Our statistic, 2.74 is relatively small, and would often be exceeded even if the marks were normally distributed. Therefore, we have no evidence that our observation shows any departure from a normal distribution.

Now for a little theory. The chance that a mark falls in the C bin in our example is p = 0.1974, and that it does not is q = 1 - p = 0.8026. We are obviously dealing with a binomial distribution for the number of marks in the bin. Now, the normal distribution is a good approximation to the binomial if the numbers are large enough, so we expect that the number of marks in a bin is normally distributed with mean E and standard deviation √E, as an approximation to the binomial distribution. Note that this means that the mean number in a bin should exceed 5 in order that this approximation be valid. Hence, (O - E)/√E is a normal variate with mean zero and unit standard deviation. The sum of the squares of such a variate is known to obey a certain distribution called the χ2 distribution, related to the gamma function, which depends on the number of terms in the sum. From this distribution, we can find the chances that our statistic exceeds any given value. The number of independent variates in the sum is the number of degrees of freedom. The details, and the explicit form of the distribution, can be found in texts on mathematical statistics.

The χ2 test is so useful that it is included, in several forms, in Langley. It can be applied to two samples, to see whether the differences in the proportions in the two samples suggests that the underlying proportions are different in the two cases. This gives a 2 x N table instead of a 1 x N table, and can be extended to more than two samples. The number of degrees of freedom in an M x N table is usually (M - 1)(N - 1). The value of χ2 is found exactly as above, by summing (O - E)2/E for each cell in the table. The expected values are found by assuming that the observations are distributed within each sample just as they are in the whole of all the samples.

The most important things to remember to get a valid χ2 test are that the expected values are not too small in any bin (certainly 5 or more), and that the degrees of freedom are properly evaluated. Unless you have a very large amount of data, the test is not very sensitive and errs on the side of safety. If you get a significant result, however, it is not likely to be wrong.

For reference, here is a small table of the values of χ2 for 1% probability. The top row contains the number of degrees of freedom.
1 2 3 4 5 6 7 8 9 10 11 12
6.63 9.21 11.34 13.28 15.09 16.81 18.48 20.09 21.67 23.21 24.73 26.22

References

  1. R. Langley, Practical Statistics For Non-Mathematical People (London: Pan Books, 1968 and Newton Abbot: David and Charles, 1971) pp. 269-291.
  2. C. E. Weatherburn, A First Course in Mathematical Statistics (Cambridge: C.U.P., 1968), Chapter IX.


Return to Economics Index

Composed by J. B. Calvert
Created 19 November 2000
Last revised