How to crawl a quarter billion webpages in 40 hours More precisely, I crawled 250,113,669 pages for just under 580 dollars in 39 hours and 25 minutes, using 20 Amazon EC2 machine instances. I carried out this project because (among several other reasons) I wanted to understand what resources are required to crawl a small but non-trivial fraction of the web. In this post I describe some details of what I did. Of course, there’s nothing especially new: I wrote a vanilla (distributed) crawler, mostly to teach myself something about crawling and distributed computing. Still, I learned some lessons that may be of interest to a few others, and so in this post I describe what I did. What does it mean to crawl a non-trivial fraction of the web? Code: Originally I intended to make the crawler code available under an open source license at GitHub. There’s a more general issue here, which is this: who gets to crawl the web? I’d be interested to hear other people’s thoughts on this issue. Architecture: Here’s the basic architecture:
Gallery Grabber QED Gallery Grabber QED is a tool for downloading graphic files from web based picture galleries to your hard-drive. Drag a gallery page link from your browser to Gallery Grabber's main interface to download gallery images. For Safari and Firefox users, Gallery Grabber browser extensions are available for even easier grabbing. Gallery Grabber can automatically determine the type of web gallery that has been dropped, extracting only the gallery images themselves - leaving banners, thumbnails and page design behind. When more control is needed Gallery Grabber's automatic behaviour can be overridden, forcing a gallery to be downloaded using a specific method: Gallery Page - a single webpage with large gallery images embedded within the page Thumbnail Picture Gallery - a single webpage with small to medium thumbnail images which lead to larger gallery images Thumbnail Page Gallery - a webpage with small to medium thumbnail images which lead to other webpages with embedded gallery images
DeepVacuum - Download Utility DeepVacuum Useful download utility based on wget command line tool. Program includes a vast number of options to fine tune your downloads through both http and ftp protocols. Allows users to download: complete single pages, entire sites, ftp catalogs, link lists from a text file, pictures, music, clips and more. Localized in languages: Chinese, English, German. Screenshots Expanded View Main View Log View Install wget Features Downloads entire sites or pages with all required content. Awards MaMUGs - "Drag this file into your Applications folder" Program Requirements DeepVacuum requires Mac OS X 10.4 or newer. Download DeepVacuum