News Currents: Taming Big Data

(Inside Science) -- Recently, Inside Science News Service published two stories on 3D printing (here are links to the first and the second). The technique represents a potentially fundamental change in manufacturing.

But for this week's News Currents, I wanted to bring to readers' attention another massive technological change. It's something you're probably vaguely aware of, and it has recently gained momentum in media coverage and also a name: Big Data.

Big Data represents a huge phenomenon -- not just because of what it promises to businesses and data analysts, but because the data is, ahem, so big.

This is also an important issue in science. Consider that at the Large Hadron Collider in Europe (the world's largest particle accelerator), just one of the several detectors (called Atlas) generates 23 petabytes of data each second of operation. That is 23 million gigabytes per second (for comparison, entry-level iPads and iPhones have 16 gigabyte capacities). Not all of that data is stored, but it is pretty difficult to fathom.

Big Data is changing astronomy, too, according to this article from The Atlantic.

Some colleges are changing their structure in order to bring biology majors into closer contact with the mathematical and statistical demands now crucial to the field. Warwick Arden, the provost of NC State, was quoted in this article: “This whole issue of big data science, big data analysis, is huge,” Arden said. “It doesn’t matter what field you’re in, but if you’re in the biological sciences, geneticists are producing massive quantities of data that need to be analyzed."

Gordon Bell began an experiment in 1998, attempting to archive his entire life -- read about his book here.

I covered the sports angle of big data in my coverage of the Sloan Sports Analytics Conference in March. Systems can track all the players on the field or court and show how they respond during plays, amassing millions of data points per game or match.

The business aspects are breathtaking. Every online move is tracked -- every click, every tendency. Not just the products a consumer buys, but what their friends and friends' friends buy. Add dizzying arrays of additional detail, and companies can fine tune the advertising that reaches customers and potential customers at previously impossible levels.

The accumulation of information highlights just how important it is to ask the right questions. Making the data give up its secrets is going to be a challenge, but the effort could enable businesses, scientists, and even amateurs to find correlations and relationships that have long escaped notice.

The U.S. government is also getting in on this action by promising a major research initiative on big data computing, as reported in the New York Times.

To round out the post, here's an interview with an historian discussing how people collected and organized information, pre-Big Data, from Pliny to present.