apache overloads the server

[developerinlondon]

for some reason my apache seems to be getting overloaded when hit by high traffic. There is a php script that does an xml search which takes about 30 seconds to complete.

Strange thing is all along we've been running this for years without any problems but recently it suddenly fell over one day and since then we reinstalled everything and put the old configurations back in. It was working fine but a few days after that we had to move the server to a new location with the same company. Ever since then we have been having this load problem with apache... the hosting people are saying there has been no change between the old connection and the new connection.

any ideas on where I can look?
apache configuration is identical.
php is identical.
java is running identical.
when there is a bit of traffic the apache falls over.

thanks

 

[jouell]

what OS are you using?


what does fallover mean? stop?

 

[developerinlondon]

Gentoo Linux.
fallover as in apache stops responding to requests.

 

[danomac]

I thought I was the only one that had this problem. I haven't spent much time trying to find the cause; mine falls over for ~60-90 seconds and then starts to serve clients again. The apache processes never actually die, but it is like they 'sleep' on me for a minute or so then resume processing.

I'm using mpm-prefork, are you? I wonder if it is a prefork issue. I haven't tried to resolve this with gentoo yet...

 

[jouell]

can you set debugging on the httpd process(es)?

 

[developerinlondon]

I am usin Prefork also, but this hadnt been an issue before.

jouell: I also enabled debugging for the http processes and now can see whats running. It appears it gets stuck running a php script that has an include file for a java servlet. but problem is it only happens when there are a lot of users on the site and it gets ok again if apache is restarted.

 

[danomac]

It's good to know it's a php issue; my logs got huge quickly that I couldn't keep logging on for a long period of time. What makes it worse is you usually can't force it to go into that state, it just happens on its own. At first I thought the database might have been involved in the process and you are the 2nd person I've heard that mention a php problem.

What version of php are you running? I'm still on 4.x and am wondering if 5.x has this problem.

 

[developerinlondon]

I am using php 5.x

but I think I found the problem.

I had a part of my code where I was running x many sqlite queries in a loop. It was taking 0.5 seconds for each query and this was bogging down the server. So I moved the queries to all get run after the loop in one go instead. This seems to have made a difference.

However I still feel this might be a bug with sqlite 2.8 that I am using with my php as earlier (long time ago) we did not have any problems and the code had queries running inside a loop.

The way I got to this result was by benchmarking my code part by part until I discovered where the time was being spent.

 

[danomac]

While I have a couple scripts that do sql in loops like you describe, they aren't called very often (maybe once in a 24 hour period.)

My server (as an example) can run for 10 days, then fall over. There really is no predictability to it. The interval isn't always 10 days either.

It sure can be frustrating.

 

[developerinlondon]

does the load gradually increase before falling over or it suddenly goes to an extreme and dies? Does it link to the busyness of the server (ie number of requests apache is serving)?

 

[developerinlondon]

Does anyone know if there is anything in linux that might cause some time lags when opening files, as I am suspecting that might be the reason why sqlite queries takes longer and longer when the server is busy with lots of users.

 

[danomac]

What file system are you using? reiserfs for example is very good with small files, and I've read it struggles with large files (ie database files.)

The requests on my server are fairly constant at 15 requests/second or so. I don't even think the load is all that high. It just falls over suddenly with no warning and refuses to serve requests (sometimes for up to 2-3 minutes.)