Leakyradiator
Technical User
I know of at least one content management system (using Cold Fusion) that allows search engines to spider their site content, even though it's behind passwords. Does anyone know how this is done? Thanks
Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations Mike Lewis on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
spidering behind passwords
- Thread starter Leakyradiator
- Start date
- Status
They check the browser agent and such... for instance they won't allow MSIE, but they will allow GoogleBot.
ALFII.com
---------------------
If this post answered or helped to answer your question, please reply with such so that forum members with a similar question will know to use this advice.
ALFII.com
---------------------
If this post answered or helped to answer your question, please reply with such so that forum members with a similar question will know to use this advice.
imstillatwork
IS-IT--Management
be carefull you don't allow anyone to visit simply by changing my firefox agent name to googlebot though...
- Status
Similar threads
