I have a simple project,
I need to mine a set of html file into a local drive from web for archival purposes.
Blackwidow, a spider, cannot handle the load, and netants, a mass downloader, cant handle either.
Is there a better tool?
I'm on my way to work right now, otherwise I'd hunt out a more specific tool for you, but here is a site that contains excellent information on spiders and bots, from off-the-shelf tools to do-it-yourself projects.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.