HTTP 1.1 contains powerful cache-control mechanisms to allow clients to place requirements on the operations of network caches. The specific method will be discussed below. (this post is specific to Apache, external links talk about using ASP, IIS, PHP, perl, CGI, Coldfusion, etc.)
Browsers...
The solution to the problem is not a programming issue. If you are using Apache, there is a module called mod_rewrite. There is a lot of information on the web on how to configure Apache, and the usage of the module.
This will allow for spidering of your site.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.