Googlebot...crawling...

[RISTMO]

Ok, so this is kinda intersting to me. About a week ago, I put a tracker on my website to see when Google gets there. It sends me an email with a url every time google crawls the site. But this is what I've gotten today:

March 31, 2005, 4:42 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/index.php[/URL]

March 31, 2005, 7:39 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/links.php[/URL]

March 31, 2005, 7:45 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/chevrole...[/URL]

March 31, 2005, 7:57 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/director...[/URL]

March 31, 2005, 8:18 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/contact_...[/URL]

March 31, 2005, 8:22 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/search.php[/URL]

March 31, 2005, 9:48 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/weekly_a...[/URL]

March 31, 2005, 9:54 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/about_us...[/URL]

March 31, 2005, 10:06 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/used_au...[/URL]

March 31, 2005, 10:36 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/financi...[/URL]

March 31, 2005, 10:52 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/special...[/URL]

March 31, 2005, 10:57 pm - Google crawled [URL unfurl="true"]http://www.genuineautos.com/pontiac.php[/URL]

First of all, why did it take over 6 hours to crawl 12 pages? But also, why did it crawl the links in random order? It didn't go from top to bottom or vice versa. It didn't go alphabetically. There was no pattern I noticed? Why's that? Wouldn't it make more sense for them to crawl one page, open all those pages linked to and log them as crawled then follow only the links on the pages just opened that it hadn't already just crawled?

Prolly not terribly important, but it's confusing me why there's no obvious pattern?

Rick

Rockwall Web Design
Arabic Music

 

[ChrisHirst]

The main SE crawlers don't follow links at all. There, another myth gone!

What happens is the spider visits, requests the URI it has been sent to, pulls the data stream, stores it in the database, then moves on to it's next scheduled URI.

The stored data (cache) is analysed and any links found are added to the crawl scheduler.


Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
Nightclub counting systems

So long, and thanks for all the fish.

 

[RISTMO]

Lol. Shows what I know. I'm just gonna stick to posting links to my site and putting keywords on the pages :-D. That's about the only thing that seems to work every time all the time ;-).

BTW has anyone heard what Google's April Fool's thing for this year is? I've been looking, but I don't see it yet.

Rick

RISTMO Designs: Rockwall Web Design
Arab Church: Arabic Christian Resources
Genuine Autos: Kaufman Chevrolet & Pontiac Dealer
Rick Morgan's Official Website

 

[RISTMO]

Maybe it's this --

http://www.google.com/googlegulp/

RISTMO Designs: Rockwall Web Design
Arab Church: Arabic Christian Resources
Genuine Autos: Kaufman Chevrolet & Pontiac Dealer
Rick Morgan's Official Website

 

[ChrisHirst]

I think the accidental one early this morning was more amusing, where every .com.au result disappeared from the results for a few hours Though the Australian SEOs haven't found it amusing.

It's fixed now BTW.



Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
Nightclub counting systems

So long, and thanks for all the fish.

 

[RISTMO]

Lol -- that .com.au thing is awesome. Google's jokes are kinda funny, but it's the "non" jokes like that that are best. Like when they launched Gmail a year ago ;-). What they should do is have the toolbars show everyone as a PR 10 for a day and have a bunch of random sites show backlinks from some various major sites. Get all the newbies going crazy on message boards talking about how their site got a PR 10 ;-). That could be a riot.

Last month Google showed one of my sites as having 700 links from wichita.edu and lanl.gov and 6 dc's were giving me a PR 7 (a few still do! link).
But better than that would be to give Yahoo a PR 0 (for real) for a day ;-). That could get some fun started :-D.

Rick

RISTMO Designs: Rockwall Web Design
Arab Church: Arabic Christian Resources
Genuine Autos: Kaufman Chevrolet & Pontiac Dealer
Rick Morgan's Official Website