Big Data: you don’t need a data scientist, you need a data plumber

Data is like water. It flows from one place to another. Big Data on the other hand is like the ocean, and you can very easily drown in it. It’s another reason why the ‘relevance paradox‘ will be the make or break of business decisions made from the informational tidal wave.

A quick Twitter exchange with Ray Wang from Constellation on information brokering lead me to the idea of data plumbing rather than what some analysts are calling for which is information brokering and data scientists. Ray posited that with big data you’ll be able to subscribe to streams like cable channels, and in a way this is already happening in your every day life: you follow people on Twitter and subscribe to their info-stream, you connect with people on Facebook and LinkedIn and do the same. However too many connections can lead to data overload, or in some cases just a load of garbage that you don’t want to read. But you want to remain connected with these people because now and again they do say something relevant.

So why a plumber, why not a scientist or even a cable engineer ?

Because data is alive; it grows, ebbs and flows as it moves and collates information. Data is in fact non-linear despite the pictures painted of information highways, it doesn’t get into it’s battered Ford Taurus and drives from A to B, it’s very much like a stream meandering from one point to the next, collecting stuff along the way, carving out new channels.

Scientists do the number crunching and make sense of the flood of information, but it’s the handy plumber that makes sure you get the information you need, when you need it, and by how much you need, routing the flow of a constant stream of acquired knowledge and making sure there are no leaks. You want to understand or make a decision on something you turn on the tap and the information flows to you, instantly. It’s not there when you don’t need it. It’s not constantly dripping and distracting you.

The scientist maintains the water tank, the plumber maintains the system itself. 

Who do you entrust your data to more ?