Robots functionality

[SjrH]

Do robots index all folders and subfolders on a server regardless, or do they only index the root pages and any pages linked to those pages?

Just wondering as I have quite a few descriptive html pages buried several subfolders down that contain keywords specific to the page but the links to them from the main page are handled via javascript.
Will these pages be indexed or do I need to change something to make them get indexed?

Thanks

 

[Foamcow]

No, the pages will only be found if they are linked to from another page that is found.

Alternately you may be able to place specific instructions for the robots to find the files by using a robots.txt file

<honk>*:O)</honk>
Foamcow Heavy Industries - Web site design in Cheltenham and Gloucester
Buy Languedoc wines in the UK

 

[SjrH]

Seems I can only disallow subfolders with the robots.txt file unfortunately.

So I guess if I were to include some <a href="url to page"> tags in the main page then the buried pages would be found and indexed?

 

[Foamcow]

Yes, they should be.

You are right. The robots protocol is an exclusive system. i.e. you can only use it to tell the crawlers what NOT to do.
I should have thought of that

Why do your links have to be javascript? If it's for rollovers, menus etc then you may be able to use CSS to do the same thing with better accessibility.



<honk>*:O)</honk>
Foamcow Heavy Industries - Web site design in Cheltenham and Gloucester
Buy Languedoc wines in the UK