Useful Link for Today.
This is a post in English.
There are different types of crawlers available at Cornell University's Web Lab. I think it will be useful.
http://www.weblab.infosci.cornell.edu/documentation.html
Below is an entry in Japanese I thought interesting about TopCoder.
