Hi,
Does anyone know if there is any way to tell which entries on our website logs are from crawlers and which are from real hosts. Some are obvious crawlers, some show as being behind an proxy server or firwall but it would be good to get a more 'official' report.
Thanks for any help you can offer.
Does anyone know if there is any way to tell which entries on our website logs are from crawlers and which are from real hosts. Some are obvious crawlers, some show as being behind an proxy server or firwall but it would be good to get a more 'official' report.
Thanks for any help you can offer.