What’s interesting to know about a web Crawler that visits my web app?

[ericnet]

In my website I track and record information of all visits. When a visit is a web browser I get and store: IP address, User-agent, Browser Name, Browser Version and Url Referrer.
But if a web crawler visits my website I know that I can get and store the User-agent, and the IP address. But can I also get other important data of a Crawler? Or User_agent and IP address are the unique interesting information about a web Crawler?

Because I suppose a web Crawler doesn’ t accept cookies, and there is no way to get the UrlReferrer. But.. And requested URL 'Request.Url'? And the User Host Name 'Request.UserHostName'? And the Reverse DNS? If so, Is it these data important about a web Crawler?

Thank you

 

[ca8msm]

What are you actually wanting to do here (i.e. what do you want to find out from the crawler)?


____________________________________________________________

Need help finding an answer?

Try the Search Facility or read FAQ222-2244 on how to get better results.

 

[ericnet]

Hi,

I am only putting some code in every page, so that when a web Crawler ‘visits’ my web app collect and store in the DB some important information about it. And later I can use that information to analyze crawler’s behaviors, traffic, etc,.. My web app isn’ t still public, and I know very few things about Crawlers. So, I only want to collect the most useful and common data about web crawlers visiting a web site, and that every web administrator must know at least.

 

[ca8msm]

What I'm trying to find out is what you consider "important". You can easily track which crawler visited your site, when it visited and what was requested. Is that the type of information you are after?


____________________________________________________________

Need help finding an answer?

Try the Search Facility or read FAQ222-2244 on how to get better results.

 

[ericnet]

What I'm trying to find out is what you consider "important". You can easily track which crawler visited your site, when it visited and what was requested. Is that the type of information you are after?

For example..

So, how you would get these data?:

'which crawler visited your site': ....
'when it visited': ....
'what was requested': .....

I am only asking which data about a crawler is important FOR YOU when analyzing traffic patterns in your website.

By the way, I suppose that a web Crawler only can request a web page and scan it, so, a web crawler can’ t submit forms, nor click links, nor start a session with a cookie, etc… So, perhaps the only interesting information I can get of a Crawler is ‘Who’, ‘When’ and ‘What’ requested.. As you said. Is all I said correct for you? And also for who read this post..?

Thanks