How to track Crawler Hits ? 2

[mobajwa]

I have an apache test webserver installed and was wondering if it were ever visited by a search engine crawler / spider ( what do u call it anyways !!! )

Is there a way i could tell just by opening the logs or anythign.Does anyone know the ip address of any of the major crawlers ( google preferably )

-
mobajwa

 

[lgarner]

Yes. The user-agent string typically tells you what it is.

A couple from my access_log:
Yahoo! Slurp
Googlebot/2.1
Surveybot
Nutch

I use AWStats, which breaks out and totals the crawler hits.

 

[mobajwa]

i dont have perl mod on my apache. Will get that today and then give awstats a shot. Seens the screenshots and its very impressive.
Thanx

-
mobajwa

 

[seanradio]

I use bbclone - it's works great & it's free -

Easy to install -

http://bbclone.de/

 

[seanradio]

Try this :

http://bbclone.de/

It's free and works with wonders.

 

[mobajwa]

I was actually looking for somethign that didnt need perl as i dont have it on my webserver. I found webalizer ( i think thats the name ) but your solution seems to be better.
Thanx again

-
mobajwa

 

[IamStang]

mobajwa

Before you install awstats, do a google for information concerning a hack/bacdoor that uses awstats. I have noticed several entries in my error logs reporting people trying to exploit this on my server. I do not use awstats, so no worries for me.

http://www.google.com/search?hl=en&q=awstats+hack

That link will bring up a large number of links to discussions, etc. about the hack.

From what I have read, which wasn't much, the hack is easily subverted by using the latest version of awstats. However, you may want to read a bit more about it first.

IamStang

 

[mobajwa]

thanx for the valuable information. u know what the default direcoctory for awstats is .. i have deleted the awstats forlder.. just windering if there is any more folders where it adds its files ?

-
mobajwa