Blogging the Bookshelf

Blogging my bookshelf – one book at a time

Blogging the Bookshelf header image 2

More Data Means More Noise – “The Signal and the Noise: The Art and Science of Prediction”, Nate Silver

November 21st, 2012 · No Comments · Data, Prediction, Statistical Inference, Statistics

As there is an exponential increase in the amount of available information, there is likewise an exponential increase in the number of hypotheses to investigate. For instance, the U.S. government now publishes data on about 45,000 economic statistics. If you want to test for relationships between all combinations of two pairs of these statistics—is there a causal relationship between the bank prime loan rate and the unemployment rate in Alabama?—that gives you literally one billion hypotheses to test.* But the number of meaningful relationships in the data—those that speak to causality rather than correlation and testify to how the world really works—is orders of magnitude smaller. Nor is it likely to be increasing at nearly so fast a rate as the information itself; there isn’t any more truth in the world than there was before the Internet or the printing press. Most of the data is just noise, as most of the universe is filled with empty space.

Tags: ····

No Comments so far ↓

Like gas stations in rural Texas after 10 pm, comments are closed.