Making a page invisible to spiders

[Sadruddin]

I want to do just that.

I need a page - not a part of the integral structure of my site - to NOT be picked up by spiders.
It won't have referral links in or out from my basic site structure, but it hosts documents for a yahoo mail list and we don't want it picked up by browser spiders.

Any ideas?

Thanks

 

[BabyJeffy]

I would suggest you not put it on the internet. That's the only way you can be sure it will not be indexed. Why not just put the pages you want behind a directory that requires a user/pass? All ISPs offer this - if yours doesn't, then maybe it's time to move?

Cheers,
Jeff


[tt]Jeff's Page [!]@[/!] Code Couch

http://www.codecouch.com/jeff/blog/

http://www.coedit.co.uk/

[/tt]

Make sure your web page and css validates properly against the doctype you have chosen - before you attempt to debug a problem!

FAQ216-6094

 

[feherke]

Hi

we don't want it picked up by browser spiders.

Browsers or spiders ? Spiders or bots mostly obeys to the robots exclusion standard. For the request in the subject is enough to put this in the document root :

User-agent: *
Disallow: /

For browsers, see Jeff's suggestion.

Feherke.

http://rootshell.be/~feherke/

 

[ChrisHirst]

If it simply exists on your site with no links to it crawlers won't find it.


Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.

 

[feherke]

Hi

And those pages must not contain any external link.

Otherwise the bots may find it in other site's referrer log if it is published. I saw such situation.

Feherke.

http://rootshell.be/~feherke/