shadowcrawlerframework

Modular framework for building real-world web crawlers in Python