Perl Getstore Problem

[chitownclone]

I've written a perl spider to do some analysis work on the web. The program I've written successful pulls the source code for 95% of the 10k+ URLS, but there are a select few that I get the 'getstore' function to work on.

These web sites are anything specical at all, so I am not sure why it works most of the time but not for these sites...any suggestions. Below are some sites that won't spider.

Won't Work:

http://www.THECAUSE4U.COM

http://www.TREATSERVICES.COM

Will Work:

http://www.SCAMDEFENSE.COM

http://www.SCANGINEER.COM

http://www.SCATTEREDREMNANT.COM

http://www.CHERISESPLACE.COM

Any ideas??? Thanks

 

[missbarbell]

Looking at the HTML headers the following could give you a clue:

<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="Thu, 1 Jan 1970 01:00:00 GMT">

getstore may be honouring the no-cache status of the page.

Barbie
Leader of Birmingham Perl Mongers

http://birmingham.pm.org