Ten Fatal Flaws in Data Analysis | Stats With Cats Blog 1. Where’s the Beef? In a way, the worst flaw a data analysis can have is no analysis at all. Instead, you get data lists, sorts and queries, and maybe some simple descriptive statistics but nothing that addresses objectives, answers questions, or tells a story. 2. If there were to be a fatal flaw in an analysis, it would probably involve how well the samples represent the population. 3. Sometimes the population is real and well defined, but the samples don’t represent it adequately. 4. The number of samples always seems to be an issue in statistical studies ( 5. Most people don’t appreciate variance. 6. NASA uses checklists to ensure that every astronaut does things correctly, completely, and consistently. 7. If a statistical test is conducted in a study, false positives and false negatives can be controlled, or at least, evaluated. 8. Here’s where you have to use your gut feel. 9. 10. Any Questions? Like this:
Google NGram Experiments With Google’s new tool Ngram Viewer, you can visualise the rise and fall of particular keywords across 5 million books and 500 years! See how big cocaine was in Victorian times. The spirit of inquiry over the ages. The spirit of inquiry over the ages II (NGram is case-sensitive). The Battle Of The Brains What happened around 1700??? Age-old debates (by Andy, James Rooney, Nick, Bidzubido, Jacqui,Gary,Stefan Lasiewski,Mark) Got any more? Rencontre avec David McCandless » Article » OWNI, Digital Journalism Le journaliste du Guardian tient le site "Information is beautiful", sur lequel il met en scène toutes sortes de données. Entretien autour des problématiques que pose la visualisation de données. Boire un thé avec David McCandless d’Information is beautiful quand on s’intéresse à la visualisation de données revient un peu à partager un pétard avec ses rockers préférés quand on est une groupie. Work In progress Là, il me montre une infographie sur les exoplanètes qu’il termine actuellement pour The Guardian. “J’ai vraiment voulu prendre le temps de sélectionner les informations pertinentes afin de créer une bonne histoire mais aussi de trouver l’échelle adéquate pour rendre le tout compréhensible.” La notion d’échelle est fondamentale pour moi ; je crois que c’est véritablement la clé de la visualisation de données car elle donne à la fois le contexte et le sens. La genèse “C’est alors que j’ai commencé à dessiner un schéma, pour faire le point et m’y retrouver. Le design et la publicité
Notabilia – Visualizing Deletion Discussions on Wikipedia The Joy of Stats About the video Hans Rosling says there’s nothing boring about stats, and then goes on to prove it. A one-hour long documentary produced by Wingspan Productions and broadcast by BBC, 2010. A DVD is available to order from Wingspan Productions. Director & Producer; Dan Hillman, Executive Producer: Archie Baron. ©Wingspan Productions for BBC, 2010 The change from large to small families reflects dramatic changes in peoples lives. Hans Rosling asks: Has the UN gone mad? Hans Rosling explains a very common misunderstanding about the world: That saving the poor children leads to overpopulation.
Data journalism pt5: Mashing data (comments wanted) This is a draft from a book chapter on data journalism (part 1 looks at finding data; part 2 at interrogating data; part 3 at visualisation, and 4 at visualisation tools). I’d really appreciate any additions or comments you can make – particularly around tips and tools. UPDATE: It has now been published in The Online Journalism Handbook. Mashing data Wikipedia defines a mashup particularly succinctly, as “a web page or application that uses or combines data or functionality from two or many more external sources to create a new service.” This ‘match’ is typically what makes a mashup. Why make a mashup? Mashups can be particularly useful in providing live coverage of a particular event or ongoing issue – mashing images from a protest march, for example, against a map. Some web developers have built entire sites that are mashups. Finally, mashups offer an opportunity for juxtaposing different datasets to provide fresh, sometimes ongoing, insights. Mashup tools Yahoo! Mashups and APIs
Data journalism pt4: visualising data – tools and publishing (comments wanted) This is a draft from a book chapter on data journalism (here are parts 1; two; and three, which looks the charts side of visualisation). I’d really appreciate any additions or comments you can make – particularly around tips and tools. UPDATE: It has now been published in The Online Journalism Handbook. Visualisation tools So if you want to visualise some data or text, how do you do it? The best-known tool for creating word clouds is Wordle (wordle.net). ManyEyes (manyeyes.alphaworks.ibm.com/manyeyes/) also allows you to create word clouds and tag clouds – as well as word trees and phrase nets that allow you to see common phrases. More general visualisation tools include widgenie (widgenie.com), iCharts (icharts.net), ChartTool (onlinecharttool.com) and ChartGo (www.chartgo.com). If you want more control over your visualisation – or want it to update dynamically when the source information is updated, Google Chart Tools (code.google.com/apis/charttools) is worth exploring. Like this:
Data journalism pt3: visualising data – charts and graphs (comments wanted) This is a draft from a book chapter on data journalism (the first, on gathering data, is here; the section on interrogating data is here). I’d really appreciate any additions or comments you can make – particularly around considerations in visualisation. A further section on visualisation tools, can be found here. UPDATE: It has now been published in The Online Journalism Handbook. “At their best, graphics are instruments for reasoning about quantitative information. Visualisation is the process of giving a graphic form to information which is often otherwise dry or impenetrable. Broadly speaking there are two typical reasons for visualising data: to find a story; or to tell one. In the parking tickets story above, for example, it was the process of visualisation that tipped off Adrian Short and Guardian journalist Charles Arthur to the story – and led to further enquiries. In most cases, however, the story will not be as immediately visible. Types of visualisation Like this:
Data journalism pt2: Interrogating data This is a draft from a book chapter on data journalism (the first, on gathering data, is here). I’d really appreciate any additions or comments you can make – particularly around ways of spotting stories in data, and mistakes to avoid. UPDATE: It has now been published in The Online Journalism Handbook. “One of the most important (and least technical) skills in understanding data is asking good questions. An appropriate question shares an interest you have in the data, tries to convey it to others, and is curiosity-oriented rather than math-oriented. Visualizing data is just like any other type of communication: success is defined by your audience’s ability to pick up on, and be excited about, your insight.” Once you have the data you need to see if there is a story buried within it. The first stage in this process, then, is making sure the data is in the right format to be interrogated. Insert: Cleaning up data Some tips for cleaning your data include: Like this: Like Loading...
Data journalism pt1: Finding data (draft – comments invited) The following is a draft from a book about online journalism that I’ve been working on. I’d really appreciate any additions or comments you can make – particularly around sources of data and legal considerations The first stage in data journalism is sourcing the data itself. Often you will be seeking out data based on a particular question or hypothesis (for a good guide to forming a journalistic hypothesis see Mark Hunter’s free ebook Story-Based Inquiry (2010)). There are a range of sources available to the data journalist, both online and offline, public and hidden. national and local government;bodies that monitor organisations (such as regulators or consumer bodies);scientific and academic institutions;health organisations;charities and pressure groups;business;and the media itself. One of the best places to find UK government data online, for example, is Data.gov.uk, an initiative influenced by its US predecessor Data.gov. Private companies and charities Live data Books and FOI