My Linux Web Server keeps crashing, Can someone help me solve this?

[nichiai]

I have a problem that has been troubling me for about 2-3 months now and I havent been able to figure out what is going wrong. This is my first post so please be kind. Usually I'm able to find the solution to my problems myself, but with this one being so general, I don't know how to begin searching for the solution.

Basically, my debian server crashes every couple of days. It doesnt crash as in restart or shutdown completely, but just so slow that it cannot be used. Some examples,

I try to attempt to load a website on the server. The first page loads (most likely because its cached), but then once I try to click on a link or login it just sits attempting to login until a time out response is given.

If I attempt to login via SSH, sometimes I cant even log in, it just sits thinking forever and ever. Other times I am able to get in at least, and then when I attempt to check what is going on, for example I run: top, it clears the screen as it would before displaying the information, but just stays at a black screen and is unable to load the information.

If I am able to log in, then sometimes I attempt to reboot the server, and I can't even do that. It will start to reboot but then just hang completely at different things such as mysql, or apache, or whatever, and I will have to manually shutdown the PC, by holding down the power button.

Manually shutting down the PC seems to be the only way to restart the server, and once it reboots everything is top speed and in working order until the next day or so when it slows down to a halt and requires a reboot.

PLEASE SOMEONE HELP ME, WHAT SHOULD I DO? If you need any log files or anything please let me know. The times in which the slowdown occurs seem a bit random, sometimes usually after the weekends on Monday morning, and on Tuesday morning. But its really random. And there is nothing happening in terms of cron jobs on that dates that its happening.

I hope someone can help me

 

[IPGuru]

It souds liake a posibible DoS attack to me

Tyr disconecting the server from the network for a few minuetes next time it happens & see if things recover

resolving the attack is unfortunatly beyond my abilities

 

[nichiai]

Thank you for your comments IPGURU. It would surprise me if it was a DoS attack, as this is a small insignificant server that we only use to run our internal company websites on and display a couple pre production websites to our clients. We are a small company and only have a handful of clients, so I can't imagine anyone have great interest to mess with us. But lets assume for the time being that it is a DoS attack, how can i verify its a DoS attack?

We currently have this server located behind a heavy duty firewall built for roughly 500 users and its just our small office of 10 users. Please let me know where to look, I have no problem researching, i just have no idea the direction I should go any longer. Im not the biggest expert on linux, so dont know how to check most diagnostic things.

thanks

 

[RhythmAce]

I agree with you that since you are a small server, there isn't much chance of it being DoS attack. But to answer your question, a real good way to tell is to look at your network traffic. If your router has lights and they're blinking like crazy, you either all of a sudden got very popular or it is indeed a DoS attack. I'm going to guess that iyt is a memory issue of some kind. This can be anything from a very poorly written script that is snatching all your memory and not releasing it or a server that keeps going into a death spiral. That is where it just keeps creating child processes. Sometimes a database server can use a lot of memory especially if you allow persistant connections. Sorry for not being more specific but I'm just trying to give you a few places to look. One thing in particular though would be your virtual memory. If a large percentage of it is being used, you can bet your bottom dollar that something is using up all your real memory. You could also have some bad memory. If you can at least narrow it down to a hardware or software problem, we can get a little more specific.

 

[haykatyck]

You can use tcpdump, Ethereal, Wireshark to see what traffic is hitting your server. I agree that since it is internal, this probably isn't it.

Does debian have "top"? If so, it will show you the processes you are running and how much CPU, RAM, what the load on teh system is, etc. Open a Terminal, if you are using a GUI, and type 'top' and press enter. Pressing 'q' will quit the application. Is should refresh itself every 5 seconds or so.

You can also use 'ps -ef' and look through the process to see what has the most CPU time.

RhythmAce has a good point too, look for defunct or zombie processes where an application or script hasn't cleaned up itself properly.

Most Live CD's have a hardwarechecker bult into them as well; when the Linux prompt comes up, you can type 'memtest' and have it check the physical RAM as well.

John
<lots of meaningless certifications here>

 

[macd68]

Install the sysstat tools if they are not already installed.

http://pagesperso-orange.fr/sebastien.godard/

For more accessible (read remote) solutions you could also use snmp to monitor the entire service envelope from process overhead and system load + various enterprise mib info to network i/o, etc.. or use something like hyperic or nagios (though I don't like nagios and wouldn't seriously recommend it.)

http://www.hyperic.com/

Monitoring is the single best way to find out what's going on.

 

[jgunkel]

The sar utilities are also good for sleuthing stuff like this. Start the sa daemon to collect per minute (configurable) stats that you can then use sar report on both while the machine is spiraling down and after the reboot.

That will give you memory usage, disk activity, process counts, etc.

It would be worth it to do this even if you have top open in a terminal somewhere, as the sa information will still be there to see even after the server is rebooted. Since this is recurring, this will, at the very least, be able to pinpoint a few spots to look at in your logs.

Good Luck!

 

[macd68]

Ummm...sysstat in the sar included application suite.

 

[macd68]

Sorry..sar is in the sysstat application suite. Too fast with the draw.