error/access log entry, what is happening here?

[Tracey]

Hi,

I have the following entries in my error.log and access.log.

I understand that the robot.txt entries are web crawlers, no problem. I am, however concerned that this crawler seems to somehow know about my .exe (which in in cgi-bin) which is a private in-house web application. This crawler appears to be trying to run/access my app??

1. how does it know about my application and related .js files??

2. what is it trying to do?

access.log:

crawl25-public.alexa.com - - [10/Aug/2003:00:56:41 +1200] "GET /robots.txt HTTP/1.0" 404 772
crawl25-public.alexa.com - - [10/Aug/2003:00:56:42 +1200] "GET /TermsOfUse.html HTTP/1.0" 200 13512
crawl24-public.alexa.com - - [10/Aug/2003:01:07:38 +1200] "GET /robots.txt HTTP/1.0" 404 772
crawl24-public.alexa.com - - [10/Aug/2003:01:07:39 +1200] "GET /etime.js HTTP/1.0" 200 3547
crawl23-public.alexa.com - - [10/Aug/2003:01:10:57 +1200] "GET /robots.txt HTTP/1.0" 404 772
crawl23-public.alexa.com - - [10/Aug/2003:01:10:57 +1200] "GET /etrace.exe/logout?ID=0 HTTP/1.0" 404 772
crawl22-public.alexa.com - - [10/Aug/2003:01:11:51 +1200] "GET /robots.txt HTTP/1.0" 404 772
crawl22-public.alexa.com - - [10/Aug/2003:01:11:51 +1200] "GET /datepicker.js HTTP/1.0" 200 16536
crawl25-public.alexa.com - - [10/Aug/2003:01:13:02 +1200] "GET /robots.txt HTTP/1.0" 404 772

crawl25-public.alexa.com - - [10/Aug/2003:01:13:03 +1200] "GET /etrace.exe/menu?ID=0 HTTP/1.0" 404 772

crawl22-public.alexa.com - - [10/Aug/2003:01:16:36 +1200] "GET /etrace.js HTTP/1.0" 200 2408
crawl23-public.alexa.com - - [10/Aug/2003:01:23:55 +1200] "GET /etrace1.js HTTP/1.0" 200 712

error.log:

[Sun Aug 10 00:56:41 2003] [error] [client 209.237.238.175] File does not exist: C:/Apache/htdocs/robots.txt
[Sun Aug 10 01:07:38 2003] [error] [client 209.237.238.174] File does not exist: C:/Apache/htdocs/robots.txt
[Sun Aug 10 01:10:57 2003] [error] [client 209.237.238.173] File does not exist: C:/Apache/htdocs/robots.txt
[Sun Aug 10 01:10:57 2003] [error] [client 209.237.238.173] File does not exist: C:/Apache/htdocs/etrace.exe
[Sun Aug 10 01:11:51 2003] [error] [client 209.237.238.172] File does not exist: C:/Apache/htdocs/robots.txt
[Sun Aug 10 01:13:02 2003] [error] [client 209.237.238.175] File does not exist: C:/Apache/htdocs/robots.txt
looking for it in the webserver root dir?
[Sun Aug 10 01:13:03 2003] [error] [client 209.237.238.175] File does not exist: C:/Apache/htdocs/etrace.exe

thanks in advance
Tracey



Remember... True happiness is not getting what you want...

Its wanting what you have got!

 

[j1bber]

ITs simply following links, thats what a crawler does, normally to index pages and submit them to search engine databases.
The references to robots.txt are related to a robot (ie crawler) control file that stops it following certain routes.
go see | for more detail
V

http://httpd.apache.org/docs/misc/howto.html#stoprob

http://www.robotstxt.org/wc/robots.html

 

[Tracey]

but there are no links to my program... and the "link" it seems to be trying to follow is incorrect anyhow.

How does it know about etrace.exe at all??? This is my major concern.

Remember... True happiness is not getting what you want...

Its wanting what you have got!