Apache slows and then stalls...where to look for errors?

[weblamer]

Hello all.

I recently 'inherited' a rather large and busy website running on an apache server. I am well versed in programming and databases, but no so much on actual server administration.

The website normally runs fine... however every couple of days something will happen where the server will slow to a crawl and then simply crash. (This results in a sudden flurry of emails to my inbox, and a red-faced boss storming into my office)

Not knowing what is going on, I open up the web host manager and do an HTTP restart. The web then goes back up and seems to run fine for a while. Then a day or two it happens again.

So, now I need to try and track down whatever is causing the server to stall. Not having a lot of administration experience, I am not sure what I am looking for. I check the error log, and there is the 'server status' on the web host manager, as well as other things, but I am not sure what to look for that would be a 'pressure gauge' letting me know what is causing the server to slow down and then halt.

Any pointers?

 

[Noway2]

A couple of things I can think of to check:
1 - are you running out of memory?
2 - how many simultaneous connections are allowed and how many do you have?
3 - Are you running a fairly current and stable version of Apache?

The fact that you can kill and then restart Apache (presumably without restarting the entire system), suggest that it is something related to the Apache process. This is why I asked if you are running an up to date version, as the program is well debugged and we can probably rule this out as a cause.

 

[weblamer]

I am currently running apache 1.3.37
(note, actual server is not in-house, hosted by server beach)

I telnet/ssh into the server to a command prompt and i have 'top' running. A current snapshot of my memory:

----

Mem: 2074640k total, 1945580k used, 129060k free, 119492k buffers
Swap: 2096472k total, 144k used, 2096328k free, 1233808k cached

---

so that does not look too bad.

in httpd.conf, 'max clients' is set to 150. Is that low for a fairly busy site? I am not sure where to look for number of connections. Is that 'tasks'? because that appears to be more than 150:

-------------

Tasks: 174 total, 1 running, 168 sleeping, 4 stopped, 1 zombie

 

[heffstar]

Good morning,

In regards to Apache grinding to a halt, I'm also having similar problems and a similar situation.

While I haven't inherited the web server, I'm not sure the configuration is sound and I'd like to modify it to something that's "good".

We have about ~1000 employees accessing the web server at different times of the day, with maybe 500 hits an hour (I'm estimating here). All of the HTML content is dynamically generated with Perl 5.

Here are some details about the server:
3.06 DualCore Xeon
4GB RAM
FreeBSD 4.9-RELEASE-p8
MySQL 4.0.20
Apache 1.3.31
Mod_Perl 1.31

Here's what I think are the important parts of httpd.conf:

KeepAlive On
MaxKeepAliveRequests 2000
KeepAliveTimeout 15
MinSpareServers 10
MaxSpareServers 20
StartServers 20
MaxClients 500
MaxRequestsPerChild 10000

I can post an output of TOP if someone really wants to see it, but it's really just 20 Apache processes sitting around doing very little with about 65MB RAM attached to each and the server sits with about 5-15MB of free RAM at any given time.

If anyone is familiar with Apache configuration and can give suggestions as to how I can free up a bunch of memory and speed this supposedly fast server, advice would be greatly appreciated.

Thanks in advance,

Hugh

 

[Noway2]

Heffstar, some of the configurations may be a little high. Specifically, the max requests at 10,000. You may also want to look into the keep alive and whether or not you want it on.

For both you and the OP, there are a few knowns that can cause Apache to run slow.
1 - reverse DNS queries being slow
2 - errors in a configuration file
3 - slow queries in a another related process, such as MYSQL
4 - being maxed out on the number of questions (make sure that you don't have an open mod_proxy) or similar.
5 - network (throughput) troubles

I would suggest starting by looking at the log files and especially the apache error log, syslog, mysql, and the daemon log for errors as these will more often than not, show if there is anything strange going on.

 

[heffstar]

Thanks for the tips Noway2; I'll look into what you suggest.

I'm thinking that most of it (and this is a somewhat educated guess) has to do with the Apache configuration since in the TOP output, it's mainly a bunch of Apache processes sitting nearly idle with a bunch of RAM allocated.

I'll continue the hunt and if I learn anything of value, I'll post it here for others with the same issues.