How often should google spider?

[spartiums]

I have a couple of question about search engine web crawlers:

When I look on Google at most of my pages on my site and click on 'Cache' it says that the snapshot was taken from 20th Oct - some on 16th Oct - shouldn't they all have been spidered on the same date? Should it be that long before a snapshot is taken - my other site has a snapshot date of 7th Nov - what determines when a snapshot is taken and when the Google spiders crawl???

Another question, is how can I see how the history of how often Google is spidering my site - or any other search engine for that matter?

Any help, greatly appreciated.

Thanks

Teeny

 

[feherke]

Hi

shouldn't they all have been spidered on the same date?

No. The robots behaviour is to take only a few pages in one visit. Recursive request could generate overload on sites built on database access.

how can I see how the history of how often Google is spidering my site

See your web server's access log. There are some log analyzers which make separate reports on robot activity. For this the log must contain the user-agent string too. Although is possible to identify robots based on remote address, this is not often used.

Feherke.

http://rootshell.be/~feherke/

 

[ChrisHirst]

Calling the SE user agents "crawlers" is a bit of a misnomer as they don't actually crawl from site to site or link to link directly.
They are sent out each time from the SE datacentre and each page has a independant schedule.



Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.

 

[mechj]

I do believe google will crawl you site more if it is being changed everyday. Think blogs, these sites are updated sometimes hourly, so google will crawl them more often, someties several times a day.

regards

jm