crawling behind a password

[Leakyradiator]

There's a content management system for online magazines, done in cold fusion and the programmers claim that search engines can crawl their articles, even though they're behind a password. Does anybody know how this is done?

Thanks

 

[Wullie]

Hi mate,

The only way I can think of to do this would be to check the browser strings and determine if the caller is a robot or not, but if that is the method used, it could be easily bypassed by anyone.

Hope this helps

Wullie

http://www.freshlookhosting.com

http://www.tutorial-world.com

http://www.freshlookdesign.co.uk

The pessimist complains about the wind. The optimist expects it to change. The leader adjusts the sails. - John Maxwell

 

[philhege]

Wouldn't every one of those crawled documents be represented as the login screen?

Phil Hegedusich
Senior Web Developer
IIMAK

http://www.iimak.com

-----------
Boy howdy, my Liberal Studies degree really prepared me for this....
-----------
A skeleton walks into a bar, and says "I'll have a beer and a mop.

 

[Leakyradiator]

The only thing I can guess at is that they identify robots in some fashion and, once identified, they establish a session for it. That would take the login out of the equation.

 

[imstillatwork]

you can download bot lists that are updated from various sources.