Access Log Format 2

[coper]

Hello,
I have rh7.1 installed hosting two of my domains. I have been trying to get one of my sites listed in google. On one of the google forums someone mentioned to look for googlebot in logs. The only thing I have seen resembling a search engine have been request for robots.txt but never see anything like googlebot or names of any other robots that have visited my server. I know googlebot has visited my server because one of my sites is listed with google. Is there something I need to configure to capture the names of search engine robots? My access log only shows ip addresses, time of request and page requested.

Thanks

 

[fluid11]

What web server are you running? Apache?

ChrisP If someone's post was helpful to you, please click the box "Click here to mark this post as a helpful or expert post".

 

[fluid11]

Place a robots.txt file in the root of the DocumentRoot. Many popular search engines search for a

http://www.somedomain.com/robots.txt

file to see what they should look at and what they shouldn't look at. To permit all robots complete access, use the following...

User-agent: *
Disallow:

This will allow a search engine to check your entire site. If you don't want a search engine in a certain part of your website, include a Disallow directive...

User-agent: *
Disallow: /private

To exclude all robots from the entire server, use the following...

User-agent: *
Disallow: /

ChrisP

If someone's post was helpful to you, please click the box "Click here to mark this post as a helpful or expert post".

 

[fluid11]

Even better, you can add your site to Google here...

http://www.google.com/addurl.html

ChrisP If someone's post was helpful to you, please click the box "Click here to mark this post as a helpful or expert post".

 

[coper]

Hi Fluid11,

Thanks for your response. I'm running apache. I have submitted to google. I visit this forum on occation

http://groups.google.com/groups?hl=en&group=google.public.support.general

I see questions on there sometime where people asked if google has crawled there site and a representative from google posted look for googlebot in your logs. I know that google has indexed my site recentley because I see the new changes in my page title and so forth but when I view my access logs I do not see the word googlebot anywhere. Google says you do not need a robots text file to be indexed. I'm just looking for engine robot names in my access logs so I can see what search engines have visited my server. As I said I never see specific names only ip addresses.

Thanks much

 

[fluid11]

Coper,

If you want to see hostnames, rather than IP addresses, then you probably need to turn on Reverse DNS lookups. You can turn this on using this directive in the httpd.conf file...

HostnameLookups on

The only problem with this is that it will degrade the performance of your server, since a reverse DNS lookup will need to be done for every request to the server.

ChrisP If someone's post was helpful to you, please click the box "Click here to mark this post as a helpful or expert post".