background preloader

Big_Data

Facebook Twitter

The Value of Big Data Isn't the Data - Kristian J. Hammond. By Kristian J. Hammond | 11:00 AM May 1, 2013 It is clear that a new age is upon us. Evidence-based decision-making (aka Big Data) is not just the latest fad, it’s the future of how we are going to guide and grow business. But let’s be very clear: There is a huge distinction to be made between “evidence” and “data.” The former is the end game for understanding where your business has been and where it needs to go. The confusion here is understandable. I would argue that what you want and what you need is to turn that data into a story. It may well be the case that you already have someone who looks at the data, builds the queries, interprets the results and writes up the report. If we’re going to really capitalize on Big Data, we need get to human insight at machine scale. For the most part, we know what we want out of the data.

In order to do this, your starting point has to be the story (or the communication) itself. Here’s an example. What Separates a Good Data Scientist from a Great One - Thomas C. Redman. By Thomas C. Redman | 9:00 AM January 28, 2013 Companies that wish to take full advantage of their data must build strong, new, and different organizational capabilities.

There is a lot to do, and data scientists are front and center. Good ones are rare. And critically, the difference between a great one and a good one is like the difference between lightning and a lightning bug. A good one can help you find relationships in vast quantities of disparate data — often important insights that you would not have gotten in any other way. Over the years I’ve had the privilege of working with dozens, maybe hundreds, of good statisticians, analysts, and data scientists. 1. But the great ones take this trait to an extreme. 2. What’s important here is that thousands of others did not see the error. Mathematics has turned out to provide a convenient, amazingly-effective language (Einstein used the phrase “unreasonably effective”) for describing the real world. 3.

This is a really big deal. 4. To Work with Data, You Need a Lab and a Factory - Thomas C. Redman and Bill Sweeney. By Thomas C. Redman and Bill Sweeney | 8:00 AM April 24, 2013 Companies that aim to score big over the long term with big data must do two very different things well. They must find interesting, novel, and useful insights about the real world in the data. And they must turn those insights into products and services, and deliver those products and services at a profit. While the two goals are mutually reinforcing, companies actually require two distinct departments. To succeed at the first, companies should set up and manage a “data laboratory,” staffed with data scientists, who question everything; a loose structure that promotes collaboration; a longer-term focus; and a culture that values creativity and the pursuit of “deeper understanding” above all else.

Companies must not confuse the separate roles. The laboratory. We want to doubly emphasize these points because promises of just the opposite are so loud. The factory. There are two keys to success here. Pundits: Stop Sounding Ignorant About Data - Andrew McAfee. By Andrew McAfee | 11:00 AM April 23, 2013 The current surge of enthusiasm around big data has produced a predictable backlash. Some of it, like Gary Marcus’s New Yorker post “Steamrolled by Big Data,” is insightful and well-reasoned (even though I have my quibbles with some of his points). This is not surprising, since he’s a neuroscientist as well as a writer, and so quite comfortable with data. Unfortunately, some other prominent commentators clearly aren’t. David Brooks has taken up big data in his New York Times column recently, and literary lion Leon Wieseltier posted last month in The New Republic about “What Big Data Will Never Explain.”

So as a public service here’s a short list, written for non-quant-jock pundits, of things to keep in mind always when writing about data and its uses. Absolute Certainty is Not the Goal (Because It’s Impossible). Data geeks desperately want to make better predictions using the seas of digital information available today. Big Data's Promise and Limitations.

Five years ago, few people had heard the phrase “Big Data.” Now, it’s hard to go an hour without seeing it. In the past several months, the industry has been mentioned in dozens of New York Times stories, in every section from metro to business. (Wired has even already declared it passé: “STOP HYPING BIG DATA AND START PAYING ATTENTION TO ‘LONG DATA’.”) At least one corporation, the business-analytics firm SAS, has a Vice-President of Big Data. Most of what’s written about Big Data is enthusiastic, like Kenneth Cukier and Viktor Mayer-Schonberger’s gushing ode “Big Data: A Revolution That Will Transform How We Live, Work, and Think,” which is currently selling briskly on Amazon, or the recent Times article on Mayor Bloomberg’s geek squad, and how “Big Data’s moment, especially in the management of cities, has powerfully and irreversibly arrived.”

As companies like Google have shown, more data often means newer and better solutions to old problems. Illustration by Joost Swarte. Models.Behaving.Badly.: Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life: Emanuel Derman: 9781439164990: Amazon.com. The Quants: How a New Breed of Math Whizzes Conquered Wall Street and Nearly Destroyed It (9780307453389): Scott Patterson. Doing Data Science: Cathy O'Neil, Rachel Schutt: 9781449358655: Amazon.com. Big Data: The Management Revolution. Artwork: Tamar Cohen, Happy Motoring, 2010, silk screen on vintage road map, 26" x 18" “You can’t manage what you don’t measure.”

There’s much wisdom in that saying, which has been attributed to both W. Edwards Deming and Peter Drucker, and it explains why the recent explosion of digital data is so important. Simply put, because of big data, managers can measure, and hence know, radically more about their businesses, and directly translate that knowledge into improved decision making and performance. Consider retailing. The familiarity of the Amazon story almost masks its power.

As the tools and philosophies of big data spread, they will change long-standing ideas about the value of experience, the nature of expertise, and the practice of management. What’s New Here? Business executives sometimes ask us, “Isn’t ‘big data’ just another way of saying ‘analytics’?” Volume. Velocity. Book Review - The Filter Bubble - By Eli Pariser. Big Data Is Great, but Don’t Forget Intuition. Andrew McAfee, principal research scientist at the M.I.T. Center for Digital Business, led off the conference by saying that Big Data would be “the next big chapter of our business history.” Next on stage was Erik Brynjolfsson, a professor and director of the M.I.T. center and a co-author of the article with Dr. McAfee. Big Data, said Professor Brynjolfsson, will “replace ideas, paradigms, organizations and ways of thinking about the world.” These drumroll claims rest on the premise that data like Web-browsing trails, sensor signals, GPS tracking, and social network messages will open the door to measuring and monitoring people and machines as never before.

The results, according to technologists and business executives, will be a smarter world, with more efficient companies, better-served consumers and superior decisions guided by data and analysis. I’ve written about what is now being called Big Data a fair bit over the years, and I think it’s a powerful tool and an unstoppable trend. Universities Offer Courses in a Hot New Field - Data Science.