background preloader

The 70 Online Databases that Define Our Planet

The 70 Online Databases that Define Our Planet
Back in April, we looked at an ambitious European plan to simulate the entire planet. The idea is to exploit the huge amounts of data generated by financial markets, health records, social media and climate monitoring to model the planet’s climate, societies and economy. The vision is that a system like this can help to understand and predict crises before they occur so that governments can take appropriate measures in advance. There are numerous challenges here. Nobody yet has the computing power necessary for such a task, neither are there models that will can accurately model even much smaller systems. But before any of that is possible, researchers must gather the economic, social and technological data needed to feed this machine. Today, we get a grand tour of this challenge from Dirk Helbing and Stefano Balietti at the Swiss Federal Institute of Technology in Zurich. These and other pursuits are now producing massive amounts of data, many of which are freely available on the web.

How To Access THE HIDDEN WIKI (Deep web, secret Internet) (Some of this was copied from, I'm just bringing it to TBN) For those who don't know, the deep web represents a gargantuan part of the internet which is not accessible through regular searches via google or other search engines. Searching on the Internet today can be compared to dragging a net across the surface of the ocean. While a great deal may be caught in the net, there is still a wealth of information that is deep, and therefore, missed. The reason is simple: Most of the Web's information is buried far down on dynamically generated sites, and standard search engines never find it.

The Palestine Papers Attack on S Sudan UN base may be 'war crime'#link# of abducted girls freed in Nigeria#link# Korea divers see bodies inside sunken ferry#link# delays Keystone XL pipeline decision#link# leaders propose constitutional change#link# Francis leads Good Friday procession#link# bomb blast kills policeman#link# Korea arrests captain of sunken ferry#link# signs law blocking Iran's envoy to UN#link# urges Israel to reverse West Bank moves#link# ailing president wins fourth term#link# Sudan deploys army to guard UN base#link# earthquake rattles Mexico#link# Ukraine's political crisis#link# Korea ferry tragedy death toll rises#link#

5 Ways to find, mix and mash your data One of the most popular trends in online journalism is taking publicly available data and translating it into visualizations or infographics that readers and viewers can quickly and easily understand. A large percentage of the visualizations you see on the web were built from scratch, which can take a considerable amount of time and effort. The following sites allow you to mash your data in record time. Swivel Public Data Sets A corpus of web crawl data composed of over 5 billion web pages. This data set is freely available on Amazon S3 and is released under the Common Crawl Terms of Use. Last Modified: Mar 17, 2014 17:51 PM GMT Three NASA NEX datasets are now available, including climate projections and satellite images of Earth.

All of our data journalism in one spreadsheet Data journalism and visualisation from the Guardian Illustration: Lisa Villani for the Guardian Today we have a list of all the data journalism produced here on the Guardian datablog. You can see what's been popular and get easy access to the data we have curated as this list shows each datablog post's associated spreadsheets, authors, the number of comments and retweets. Tools to help bring data to your journalism « Michelle Minkoff NOTE: This entry was modified on the evening of 11/9/10 to deal with typos and missing words, resulting from posting this too late the previous night. Sleep deprivation isn’t always a good thing — although it allows one to do things more fun than sleep. Like play with data.

Finding Data on the Internet Skip to Content A Community Site for R – Sponsored by Revolution Analytics Home » How to » Finding Data on the Internet Finding Data on the Internet By RevoJoe on October 6, 2011 Real-Time Data And A More Personalized Web - Smashing Magazine Advertisement As Web designers, we face a daily struggle to keep pace with advances in technology, new standards and new user expectations. We spend a large part of our working life dipping in and out of recent developments in an attempt to stay both relevant and competitive, and while this is what makes our industry so exciting to be a part of, it often becomes all too easy to get caught up in the finer details. Responsive Web design, improved semantics and rich Web typography have all seen their fair share of the limelight over the last year, but two developments in particular mark true milestones in the maturation of the Web: “real-time data” and a more “personalized Web.” Since the arrival of the new Web, we’ve been enraptured by social media.

How to access 100M time series in R in under 60 seconds DataMarket, a portal that provides access to more than 14,000 data sets from various public and private sector organizations, has more than 100 million time series available for download and analysis. (Check out this presentation for more info about DataMarket.) And now with the new package rdatamarket, it's trivially easy to import those time series into R for charting, analysis, or anything. Here's what you need to do:

World Values Survey From this section you can download the different respondent level data files, in format compatible with some popular statistical programs (SPSS, Stata or SAS).. Click over the different links to begin downloading the files. If you encounter any problems please contact us at

A computational journalism reading list [Last updated: 18 April 2011 -- added statistical NLP book link] There is something extraordinarily rich in the intersection of computer science and journalism. It feels like there’s a nascent field in the making, tied to the rise of the internet. Take the tour: Go beyond Embed tables and charts Embed any chart or table from on your own web sites, in your blog posts or news articles. Chart appearance can be configured to match your branding and other requirements. Create dashboards Build a destination for your target audience with dashboards that live on your own web site.

Tariffs: Comprehensive tariff data on the WTO website What are you looking for? Sophisticated, detailed and interactive analysis? > Go to the new Tariff Analysis Online; > Explanation and user guide: browse, pdf, Word Simpler, standardized tariff statistics, mainly for downloading? > Go to WTO Tariff Download Facility; > Explanation and user guide: browse, pdf, Word