background preloader

Systems Thinking around Big Data

Facebook Twitter

Machine Learning, Big Data, Deep Learning, Data Mining, Statistics, Decision & Risk Analysis, Probability, Fuzzy Logic FAQ. The Tenuous Relationship Between Big Data, Demographic, Behavior and Intent. I’ve a confession to make – I’m more than a little skeptical about Big Data’s capability to predict consumer behavior (and desire).

The Tenuous Relationship Between Big Data, Demographic, Behavior and Intent

I’m not saying it’s useless: far from it. I believe that it has the potential to provide some valuable information about (and insight into) the actions of consumers. IT Blog Don’t sell the technology, sell the application of big data - Insight IT Blog. Emma DeSousa May 21, 2013 Whether it’s the boardroom or a potential customer you are trying to convince, make sure you have good case study or two at hand, says Emma de Sousa.

IT Blog Don’t sell the technology, sell the application of big data - Insight IT Blog

Patterns amid Complexity. By Greg Fisher Among the people I talk to about complex systems, a common point made is that the future is inherently uncertain or “unknowable”.

Patterns amid Complexity

The End of Theory: The Data Deluge Makes the Scientific Method Obsolete. "All models are wrong, but some are useful. " So proclaimed statistician George Box 30 years ago, and he was right. But what choice did we have? Only models, from cosmological equations to theories of human behavior, seemed to be able to consistently, if imperfectly, explain the world around us. Until now. The Petabyte Age: Because More Isn't Just More — More Is Different. Big Data needs Big Theory. In this guest cross-post, Geoffrey West, former President of the Santa Fe Institute, argues that just as the industrial age produced the laws of thermodynamics, we need universal laws of complexity to solve intractable problems of the post-industrial era, and that ‘big data’ needs such ‘big theory’.

Big Data needs Big Theory

For more on this topic, see David Hales’ guest post from February this year ‘Lies, Damned Lies and Big Data’. As the world becomes increasingly complex and interconnected, some of our biggest challenges have begun to seem intractable. What should we do about uncertainty in the financial markets? How can we predict energy supply and demand? How will climate change play out? What does this mean? What makes a “complex system” so vexing is that its collective characteristics cannot easily be predicted from underlying components: the whole is greater than, and often significantly different from, the sum of its parts. Harnessing Big Data with a Systems Thinking Approach – (A Harley Davidson Case Study)

With 90% of the world’s data created in the last two years, what can we expect our data vaults to hold two or even twenty years from now?

Harnessing Big Data with a Systems Thinking Approach – (A Harley Davidson Case Study)

Today we measure our lives in peta-bytes but by 2020 estimates show a 2,300% increase in the bits and bytes that will define our lives. 35 zeta-bytes to be exact. How then can we as a society leverage the intrinsic value of so much data without getting bogged down with its complexity? Around the turn of the century, we experienced a similar moment of euphoria when retail outlets opened ‘virtual stores’ and sold products to online buyers. A famous IBM TV ad once depicted an overwhelmed young company whose products went from a few online orders a day to hundreds of thousands. The Future of Decision Making: Less Intuition, More Evidence - Andrew McAfee. By Andrew McAfee | 9:00 AM January 7, 2010 Human intuition can be astonishingly good, especially after it’s improved by experience.

The Future of Decision Making: Less Intuition, More Evidence - Andrew McAfee

Savvy poker players are so good at reading their opponents’ cards and bluffs that they seem to have x-ray vision. Firefighters can, under extreme duress, anticipate how flames will spread through a building. And nurses in neonatal ICUs can tell if a baby has a dangerous infection even before blood test results come back from the lab. The lexicon to describe this phenomenon is mostly mystical in nature. Prediction. The purpose of Intelligence is Prediction.


Evolution of the ability to predict agents and phenomena in the environment improved survival rates and created a strong evolutionary pressure to develop better and longer term predictions. This is the reason Intelligence evolved. These paragraphs are discussing a hypothesis of how Intelligence may have evolved but you should know that my knowledge of Evolutionary Biology is limited. The earliest brains might have been nothing more than simple nerve clusters that coordinated the movement of legs while walking in multi-legged agents, such as arthropods.

Walkers keep some of their legs on the ground at all times which means you only lift one or a few of them at any one time. When using a "running gait" all legs will be off the ground at the same time for some part of the running cycle. In order for a feature to be Biologically Plausible there must exist a way that feature could evolve. Vision evolved before running. Intuition versus Logic. Most humans have not been taught logical thinking, but most humans are still intelligent.

Intuition versus Logic

Contrary to the majority view, it is implausible that the brain should be based on Logic; I believe intelligence emerges from millions of nested micro-intuitions, and that Artificial Intelligence requires Artificial Intuition. Intuition is surprisingly easy to implement in computers. Intuition and Logic are two strategies for prediction and problem solving.