background preloader


Facebook Twitter

An Introduction to Compassionate Screen Scraping. Screen scraping is the art of programatically extracting data from websites.

An Introduction to Compassionate Screen Scraping

If you think it's useful: it is. If you think it's difficult: it isn't. And if you think it's easy to really piss off administrators with ill-considered scripts, you're damn right. This is a tutorial on not just screen scraping, but socially responsible screen scraping. Its an amalgam of getting the data you want and the Golden Rule, and reading it is going to make the web a better place. We're going to be doing this tutorial in Python, and will use the httplib2 and BeautifulSoup libraries to make things as easy as possible.

Websites crash. For my blog, the error reports I get are all generated by overzealous webcrawlers from search engines (perhaps the most ubiquitous specie of screenscraper). This brings us to my single rule for socially responsible screen scraping: screen scraper traffic should be indistinguishable from human traffic. Cache feverently. Setup Libraries Choosing a Scraping Target Ending Thoughts. Code Like a Pythonista: Idiomatic Python. In this interactive tutorial, we'll cover many essential Python idioms and techniques in depth, adding immediately useful tools to your belt.

Code Like a Pythonista: Idiomatic Python

There are 3 versions of this presentation: ©2006-2008, licensed under a Creative Commons Attribution/Share-Alike (BY-SA) license. My credentials: I am a resident of Montreal,father of two great kids, husband of one special woman,a full-time Python programmer,author of the Docutils project and reStructuredText,an editor of the Python Enhancement Proposals (or PEPs),an organizer of PyCon 2007, and chair of PyCon 2008,a member of the Python Software Foundation,a Director of the Foundation for the past year, and its Secretary. Tail Recursion Elimination. I recently posted an entry in my Python History blog on the origins of Python's functional features.

Tail Recursion Elimination

Documentation Index. Building Skills in Python — Building Skills in Python. A Programmer’s Introduction to Python Legal Notice This work is licensed under a Creative Commons License.

Building Skills in Python — Building Skills in Python

You are free to copy, distribute, display, and perform the work under the following conditions: Attribution. You must give the original author, Steven F. For any reuse or distribution, you must make clear to others the license terms of this work. Language Basics The Processing View A programming language involves two closely interleaved topics. Basic Python Exercises - Google's Python Class - Google Code. ICPCWiki: Problem Resources. About ICPC The ACM International Collegiate Programming Contest (ICPC) is a multitier, team-based, programming competition operating under the auspices of ACM and headquartered at Baylor University.

ICPCWiki: Problem Resources

The contest involves a global network of universities hosting regional competitions that advance teams to the ACM-ICPC World Finals. Participation has grown to several tens of thousands of the finest students and faculty in computing disciplines at almost 2,330 universities from over 91 countries on six continents. The contest fosters creativity, teamwork, and innovation in building new software programs, and enables students to test their ability to perform under pressure. Quite simply, it is the oldest, largest, and most prestigious programming contest in the world. Continue with the link below. Quantitative Reasoning 20 Assignments. There will be weekly assignments listed here.

Quantitative Reasoning 20 Assignments

They will usually be assigned on Thursday and due the following Friday. Python - Notes. You have seen how you can reuse code in your program by defining functions once.

Python - Notes

What if you wanted to reuse a number of functions in other programs that you write? As you might have guessed, the answer is modules. There are various methods of writing modules, but the simplest way is to create a file with a .py extension that contains functions and variables. Another method is to write the modules in the native language in which the Python interpreter itself was written. For example, you can write modules in the C programming language and when compiled, they can be used from your Python code when using the standard Python interpreter. A module can be imported by another program to make use of its functionality. Example (save as import sys print('The command line arguments are:')for i in sys.argv: print i print '\n\nThe PYTHONPATH is', sys.path, '\n' How It Works First, we import the sys module using the import statement.

Python Warmup-1 sleep_in. Structuring Your Project — pythonguide 0.0.1 documentation. By “structure” we mean the decisions you make concerning how your project best meets its objective.

Structuring Your Project — pythonguide 0.0.1 documentation

We need to consider how to best leverage Python’s features to create clean, effective code. In practical terms, “structure” means making clean code whose logic and dependencies are clear as well as how the files and folders are organized in the filesystem. Which functions should go into which modules? How does data flow through the project? What features and functions can be grouped together and isolated? In this section we take a closer look at Python’s module and import systems as they are the central elements to enforcing structure in your project.

Structure of the Repository. Is there a way to access hardware directly in Python. Did you know that?