How to mass download/mine historical news announcement from sites?

[zetic]

Hi Everyone,

I have a simple project,
I need to mine a set of html file into a local drive from web for archival purposes.
Blackwidow, a spider, cannot handle the load, and netants, a mass downloader, cant handle either.
Is there a better tool?

Thanks,
Tom

 

[jtm111]

I'm on my way to work right now, otherwise I'd hunt out a more specific tool for you, but here is a site that contains excellent information on spiders and bots, from off-the-shelf tools to do-it-yourself projects.

http://www.botspot.com/

I've found incredible tools there.

 

[apmilano]

I bought a copy of Offline Explorer which I found on

http://www.download.com

It offers a very simple way to archive websites and includes very detailed parameters for controlling what portions of the website to archive.

I highly reccommend it.