Robot txt File

[McFestoe]

Iam realy new at website's due to being forced in to creating my own after various attempts to get a Co to do it.

Iam learnt loads at the moment and finding really intresting, and enjoying it, (do i have a mental problem all these late hours ?)

I have signed up for google webmaster tools etc, and the one that keeps throwing up an error 'not found' is a Robot txt file. I have made one and stuck it in the Public area but google stills cant see it.

Does this look right to allow full access?

<meta name="Robots" content="indexall,followall" />
<meta name="revisit" content="7 days" />

Robots txt file

User-agent: *
Disallow:

Been reading up on

http://www.robotstxt.org

is there any other good sites that explain these files.

 

[carrierquotes]

The google webmaster tools are a few days / weeks delayed at least with my sites. The robots.txt has to go into the home directory of yoursite. Like yoursite.com/robots.txt

Try to load it yourself by going to the url on your domain and if you can you're fine. Note that the robots.txt has to be written in plain text (notepad).

Marco

http://pbx-installers.com

 

[Foamcow]

Also, by default Google and other SEs will follow all links etc within your site and index all pages.

Robots is an exclusion protocol.
You only need to tell it stuff that you don't want to be indexed or followed.


<honk>*:O)</honk>

Tyres: Mine's a pint of the black stuff.
Mike: You can't drink a pint of Bovril.

 

[McFestoe]

Thanks

Does this mean i dont have to bother with a Rorots.txt file, been looking at gooles webmaster tools and it flags up a error because there isnt one.

Do i need one?

 

[McFestoe]

Forgot to mention if i type in

http://www.mywebname.co.uk/robots.txt

i get a error 404 page not found, should the robots text file be displayed in ie if you type the address in ?

 

[feherke]

Hi

Do i need one?

As Foamcow wrote, you need one only if you want to interdict the crawler's access to certain parts of your site.

If you just want to make Google's error message disappear, place and empty file with robots.txt name into your site's root directory.

And yes, to work the robots.txt file must be available for user agents. Including browsers. Including Explorer. So if it exists and is available it should be displayed.

Feherke.

http://rootshell.be/~feherke/

 

[ChrisHirst]

This,

<meta name="Robots" content="indexall,followall" />

This should be

<meta name="robots" content="index,follow" />

But as this is the default you can get rid of it

this,

<meta name="revisit" content="7 days" />

on the other hand, is utterly useless. As the only search engine to ever use it no longer exists.

Get rid of it.




Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.

 

[McFestoe]

Chris

Thanks for the advice could i ask the difference between the two versions.

<meta name="Robots" content="indexall,followall" />

<meta name="robots" content="index,follow" />

Thanks

Rich

 

[ChrisHirst]

Yep

This one
<meta name="Robots" content="indexall,followall" />

is totally the wrong syntax. There are no such values as indexall or followall

the choices are

all | none : all = index,follow : none = noindex,nofollow

follow | nofollow

index | noindex

The default if the meta does not exist in the page is for index, follow


Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.