noindex, follow tag parameter

[Jeremy21b]

Hello.

I have a sitemap that has been causing me some problems. First it kept getting ranked above the content pages that in linked to. So instead of having good titles & descriptions for my search engine listings, it would be 'Sitemap'. I added a <META NAME="ROBOTS" CONTENT="NOINDEX,FOLLOW"> tag to that page. It worked great in Google, but Yahoo seems to be having problems with that tag. The page does not get indexed by Yahoo, but the spider is not following the links. So now in Yahoo, not only does our sitemap not get listed, but all of the linked pages do not get indexed either.

Has anyone else had a similar problem? Or does anyone know a way around this? Would there be a way to apply the robots tag to only the Google spider?

 

[ChrisHirst]

don't you think you've got that a bit backwards?
If your sitemap ranks better than the pages it links to then obviously the pages need some work. killing the thing thing that gets them crawled is obviously a bad move.

the meta robot instructions are not a standard and are always negative, a DO NOT instruction, googlebot handles them ok, other bots take the fact that the tag exists as a total block and most bots ignore them.

remove the tag, let the sitemap do what it is supposed to and get ranked for terms so visitors can find your product pages, optimise the sitemap so it has title and description,
it is part of your site after all!



Chris.

Indifference will be the downfall of mankind, but who cares?

http://www.candsdesign.co.uk

A website that proves the cobblers kids adage.

http://www.cram-system.com

Nightclub counting systems

So long, and thanks for all the fish.

 

[Jeremy21b]

Hi Chris.

No I do not have anything backwards. Our SEO campaign has very successful and we rank very high for hundreds of search phrases. I do not want the sitemap to show up in searches because it does not entice anyone to click on it. I'd much rather have a content page show up with targeted titles & descriptions. The reason the sitemap shows up ahead of the pages is that it is only one click from the frontpage, while the content pages are another click away.

meta robot tags are not always negative....why would there be a noindex/index & nofollow/follow? Just the Yahoo spider doesn't follow these parameters.

I guess I should take my question to a search engine forun instead of a purely technical forum.

 

[ChrisHirst]

without the robots meta in there the default is to index and follow. therefore by adding the meta you are always telling the robot NOT to do something. the same is true of the revist-after this tells the bot NOT to return for at least x days rather that visit every x days.

luckily for some, this is ignored by just about all bots.

SEO forums I'd recommend are High Rankings or IHY

Chris.

Indifference will be the downfall of mankind, but who cares?

http://www.candsdesign.co.uk

A website that proves the cobblers kids adage.

http://www.cram-system.com

Nightclub counting systems

So long, and thanks for all the fish.

 

[xtendscott]

Are you using straight HTML or a sript language as ASP, PHP, or CFM?

If you are using a script language or have Apache web server, you can check for UserAgent "Googlebot/2.1 (

http://www.google.com/bot.html)"

if it does not contain Googlebot then give this page, else if it is Googlebot then give this one.

Is it an evil "cloak", technically could be considered cloaking but pages are delivered by UserAgent all the time based on IE, Netscape, what version and how to deal with bugs in all of them.

But this solution would seem to work for your needs.

xtendscott
Web Site Design and CF Programming