Robots.txt or meta tags?

[FesterSXS]

What are peoples thoughts on the above?

Am I right in assuming that using a robots.txt file is the standard way to instruct SE robots and that the robots meta tags are not really supported.

Tony

Spirax-Sarco - steam traps control valves heat exchangers
Sun Villa - Luxury Florida accommodation

 

[Foamcow]

I use a text file... not really thought about it much to be honest.

Foamcow Heavy Industries - Web design and ranting
Target Marketing Communications - Advertising, Direct Marketing and Public Relations
I wonder what possesses people to make those animated gifs. Do you just get up in the morning and think, "You know what web design r

 

[ChrisHunt]

Most reputable search engines will honour both methods. However, the robots.txt method is a little easier on your server if you're excluding a lot of pages from the SEs. The spider will load the site's robots.txt, and know not to request the pages that are excluded. If you go the meta tag route, the spider has to load (and you have to serve) each page before it finds out it shouldn't list it. Robots.txt also alows you to do more sophisticated things, like excluding particular engines.

Of course if, like most people, you want spiders to visit every page, you won't need either a robots.txt or any special meta tags. That's what they do by default.

-- Chris Hunt
Webmaster & Tragedian
Extra Connections Ltd