404 error on existing page!

[l310564]

Hi,

I have a website where i'm getting 404 errors on a page that exists.

All the people that are getting this error have the following User Agent Header.

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

From what i have read this is invalid due to the lack of a space between ; and 1813 and is posibly caused by AVG's new security toolbar.

My first question is that if this is invalid then would that cause a 404 error and if that is the case how do I allow this invalid UA in IIS 6 windows server 2003?

Thanks,

Hugh

If knowlege can create problems it is not through ignorance that we will solve them.

Isaac Asimov

 

[ChrisHirst]

Must admit it's the first time I've ever heard of a UserAgent string causing a 404 error.

posibly caused by AVG's new security toolbar.

Why? do you have this toolbar installed on the server?


Do you have some code running on the site that reads the UA and does something with it? Because the server does nothing with the UA string other than store it in the access logs.


Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.

 

[l310564]

Thanks for the reply chris,

No I don't have version 8 of AVG installed or have any code that reads the UA but reading this forum

http://www.webmasterworld.com/search_engine_spiders/3615360.htm

the sugestion is that it is AVG and because the UA is invalid the website rejects the user thinking it's a hacker.

To be honist i don't care what is generating the invalid UA all i care about is how to stop these 404 errors because it's blocking about 10% of my web traffic. So any ideas would be welcome.

Thanks,

Hugh



If knowlege can create problems it is not through ignorance that we will solve them.

Isaac Asimov

 

[ChrisHirst]

The WMW thread is about mod_security which is an Apache module so won't apply here unless you are running Apache on Windows.

In a standard install of IIS I don't know of anything that examines user agent strings and blocks access based on that. And, if there was, the HTTP response would be a 403 (Forbidden) rather than a 404. There are firewalls that can, but in that case there would be no request to the server.

I would be looking at the behaviour pattern of the visitors with this UA and the IP range(s) used to see if it is REAL visitors being blocked or scraper bots and the like.

other things to look at;
Is it one particular page that fails or several?
Does your site require cookies at any point, and is there any correlation between cookies/sessions and the request failures?
Does the same UA get 200 responses for other pages?

And probably lots of other things that could be seen when looking at the actual logs.


Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.

 

[l310564]

Ok I have got a bit further with this issue.

Looking in the log files I can see that for all of the errors i'm worried about have the user agent string

Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;1813)

but that user agent string also returns sccesses sometimes so it isn't purely down to this.

What is confusing me at the moment is that when it fails the sc-status = 200 indicating Success while the sc-win32-status = 123 which means The filename, directory name, or volume lable syntax is incorrect.

Why are these two thing give conflicting results?

I have included two log file entries one which shows sucess and one that shows failure.

2008-06-24 10:01:33 W3SVC1542998328 xx.xxx.xxx.xxx GET /start.aspx?CAMPAIGN=UK+Google&KEYWORDS=compare%20mortgage - 80 - 78.86.212.206 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;1813) - - 200 123 6022 222 140

2008-06-24 10:01:39 W3SVC1542998328 xx.xxx.xxx.xxx GET /start.aspx CAMPAIGN=UK+Google&KEYWORDS=mortgage%20offers 80 - 81.158.245.237 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;1813) - - 200 0 6220 209 156

Thanks,

Hugh

If knowlege can create problems it is not through ignorance that we will solve them.

Isaac Asimov

 

[l310564]

Um

This is intresting but i don't understand it......

All the failures have the whole URL including the ? in the
cs-uri-stem part of the log so that would mean that IIS is trying to find a file called start.aspx?CAMPAIGN=UK+Google&KEYWORDS=compare%20mortgage rather than start.aspx and then process the query string CAMPAIGN=UK+Google&KEYWORDS=compare%20mortgage.

Where as all the querys that work split the URL into the cs-uri-stem and cs-uri-query parts of the log correctly?

WTF

Any help at all would be very much appreciated.

Thanks,

Hugh


If knowlege can create problems it is not through ignorance that we will solve them.

Isaac Asimov

 

[ChrisHirst]

check your adwords campaign setup
I would suggest somebody set up a landing page url without the "?"

Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.

 

[l310564]

That was my first thought but everything seems to be fine besides it's the ones that contain the ? that are causing the problems.

I.e.

2008-06-24 10:01:39 W3SVC1542998328 xx.xxx.xxx.xxx GET /start.aspx CAMPAIGN=UK+Google&KEYWORDS=mortgage%20offers 80 - 81.158.245.237 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;1813) - - 200 0 6220 209 156

Is fine and it is splitting cs-uri-stem and cs-uri-query correctly in the log but

2008-06-24 10:01:33 W3SVC1542998328 xx.xxx.xxx.xxx GET /start.aspx?CAMPAIGN=UK+Google&KEYWORDS=compare%20mortgage - 80 - 78.86.212.206 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;1813) - - 200 123 6022 222 140

Is lumping the whole URL including the query string into the cs-uri-stem and leaving the cs-uri-query string blank.

Thanks,

Hugh

If knowlege can create problems it is not through ignorance that we will solve them.

Isaac Asimov

 

[ChrisHirst]

the sc-status simply means the server handled the request ok (as it would) but the OS is "blowing up" because of illegal characters.

for whatever reason it would seem that the "?" is not really a "?" but possibly an unknown character in whatever set id in use.
Both IPs in your examples are UK based BT for the 81. and O2 for the 78.

Is it always the same campaign KW that fails?

Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.

 

[l310564]

Thanks for the reply chris,

This is happening on the two sites that we drive trafic to.

Is it always the same campaign KW that fails?

No it happens accross the board.

Hugh

If knowlege can create problems it is not through ignorance that we will solve them.

Isaac Asimov