Scrap Data

< < < < <

http://code.google.com/p/freebase-parallax/

Freebase Parallax provides a new way to browse and explore data in Freebase.

freebase-parallax - New way to browse and explore data in Freebase

Parallax

This is part of a four-part series on web-scraping for journalists . As of Apr. 5, 2010 , it was a published a bit incomplete because I wanted to post a timely solution to the recent Pfizer doctor payments list release , but the code at the bottom of each tutorial should execute properly.

Coding for Journalists 103: Who’s been in jail before: Cross-checking the jail log with the court system; Use Ruby’s mechanize to fill out a form | Dan Nguyen pronounced fast is danwin

http://danwin.com/2010/04/coding-for-journalists-part-3-cross-checking-the-jail-log-with-the-court-system-use-rubys-mechanize-to-fill-out-a-form/

Coding for Journalists 102: Who’s in Jail Now: Collecting info from a county jail site | Dan Nguyen pronounced fast is danwin

http://danwin.com/2010/04/coding-for-journalists-102-collecting-info-from-a-county-jail-site/

A note about privacy : This tutorial uses files that I archived from a real-world jail website.

http://danwin.com/2010/04/coding-for-journalists-go-from-a-know-nothing-to-web-scraper-in-an-hour-hopefully/

Coding for Journalists 101: Go from knowing nothing to scraping Web pages. In an hour. Hopefully. | Dan Nguyen pronounced fast is danwin

UPDATE (12/1/2011) : Ever since writing this guide, I’ve wanted to put together a site that is focused both on teaching the basics of programming and showing examples of practical code.

Coding for Journalists 104: Pfizer’s Doctor Payments; Making a Better List | Dan Nguyen pronounced fast is danwin

Update (12/30): So about an eon later, I’ve updated this by writing a guide for ProPublica . Heed that one. This one will remain in its obsolete state.

http://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/

http://www.propublica.org/nerds/item/scraping-websites

Web-scraping is essentially the task of finding out what input a website expects and understanding the format of its response.

Chapter 4: Scraping Data from HTML

http://www.propublica.org/nerds/item/reading-flash-data

Flash applications often disallow the direct copying of data from them.

Chapter 2: Reading Data from Flash Sites

Google Refine [2] (the program formerly known as Freebase Gridworks) is described by its creators as a “power tool for working with messy data” but could very well be advertised as “remedy for eye fatigue, migraines, depression, and other symptoms of prolonged data-cleaning.” Even journalists with little database expertise should be using Refine to organize and analyze data; it doesn't require much more technical skill than clicking through a webpage. For skilled programmers, and journalists well-versed in Access and Excel, Refine can greatly reduce the time spent doing the most tedious part of data-management.

Development of an automated climatic data scraping, filtering and display system 10.1016/j.compag.2009.12.006 : Computers and Electronics in Agriculture | ScienceDirect.com