Big Data is, but not unquestioningly, the answer, he says


“Big Data,”  a phrase that’s often said aloud with lots of vim (“Big Data!”), is just a heap of junk — until you know what you’re looking for. Jake Porway, writing in the Harvard Business Review, points this out more clearly than many others have:

“We have a lot of data, but we have no idea what we should do with it.” The director of the foundation looked plaintively across the table at me. “We were thinking of having a hackathon, or maybe running an app competition,” he smiled. His co-workers nodded eagerly. I shuddered.

I have this conversation about once a week. Awash in data, an organization — be it a healthcare nonprofit, a government agency, or a tech company — desperately wants to capitalize on the insights that the “Big Data” hype has promised them. Increasingly, they are turning to hackathons — weekend events where coders, data geeks, and designers conspire to build software solutions in just 48 hours — to get new ideas and fill their capacity gap….

Any data scientist worth their salary will tell you that you should start with a question, NOT the data. Unfortunately, data hackathons often lack clear problem definitions. Most companies think that if you can just get hackers, pizza, and data together in a room, magic will happen.

The same kind of confusion happened in the 1970s and 1980s, when “software” was often billed as a magical solution. Especially if it was called “artificial intelligence,” software was said to be so powerful that it would figure out the answers to big problems, no questions asked. Over time, people tended to learn that software is an engineer’s tool, not a genie’s magic lamp.

The same could be true for Big Data.

Porway founded and directs DataKind, an organization that tries to help tease out answers — answers to people’s specific questions — from big heaps o’ data. Datakind is based in New York, with outposts in Bangalore, Dublin, San Francisco, Singapore, the UK, and Washington DC.

This video shows Porway and some of his fellow Datakindpersons effusing about what Big Data can (and by implication, probably can’t and shouldn’t be asked to) do:

By the way: Pizza itself can be a Big Data question. That’s one of the messages in a December 2012 report, in PMQ Pizza Magazine, called “Pizza Power 2013 State of the Industry Report.”

Also by the way, though at a greater distance and in a different direction: sometimes Big Data is not just a collection of numbers. Sometimes Big Data is a group of people who make music.

Marc Abrahams is the editor of the Annals of Improbable Research magazine and organizer of the Ig Nobel Prizes.
Follow Marc on Twitter