Venture capital sees big returns in big data. What's the big deal about Big Data? Every so often a term becomes so beloved by media that it moves from “instructive” to “hackneyed” to “worthless,” and Big Data is one of those terms. When I started IA Ventures in 2009, I used “Big Data” as a quick and easy way to describe the thematic focus of my fund. And it worked. People got that it implied tools for managing large amounts of data and applications for extracting value from that data. It also implied a high level of technical and product expertise with specific relevance for assessing, investing in and supporting these kinds of companies. And perhaps most importantly, people understood that this wasn’t a cyclical theme but a secular phenomenon, one that would eventually touch every business and every consumer, wherever they may be.
Taken together, pursuing this market opportunity in such a focused manner resonated with corporations, LPs and entrepreneurs alike. But since this time the term Big Data has become diluted. The year in big data and data science. Big data and data science have both been with us for a while. According to McKinsey & Company’s May 2011 report on big data, back in 2009 “nearly all sectors in the U.S. economy had at least an average of 200 terabytes of stored data … per company with more than 1,000 employees.”
And on the data-science front, Amazon’s John Rauser used his presentation at Strata New York (below) to trace the profession of data scientist all the way back to 18th-century German astronomer Tobias Mayer. Of course, novelty and growth are separate things, and in 2011, there were a number of new technologies and companies developed to address big data’s issues of storage, transfer, and analysis. Important questions were also raised about how the growing ranks of data scientists should be trained and how data science teams should be constructed.
With that as a backdrop, below I take a look at three evolving data trends that played an important role over the last year. The ubiquity of Hadoop Related: Linked Data Marketplaces. Data 2.0 Conference : les startups à suivre. Is there a disconnect between Big Data and the Web of Data ? ‘Big Data‘ is currently capturing the imagination, attracting hype, investment and ambitious startups in almost equal measure. Kim and Eric Norlin’s excellent Defrag and Glue events have gained big-name company, with O’Reilly‘s Strata and GigaOM‘s Structure both set to arrive in the first quarter of 2011. Venture firms like IA Ventures have emerged, specifically targeted at finding, funding, and profiting from the big Big Data idea. Giants of the web from Yahoo! And Amazon to Twitter and Facebook solve their own Big Data problems in very different ways, contributing valuable code and experience to the community whilst simultaneously diluting focus and adding to the cacophony.
Flippantly reckoned by many to be ‘anything that requires more than a single machine to run,’ the Big Data reality remains somewhat harder to pin down. And there, for many enterprises, lies the problem. This is enough for now, though. Related Strata Conference 2011, Day 2 Keynotes In "Big Data"