high memory usage on apache

[CGI101]

ever since about 3 days ago, the server has been having VERY high memory usage which is very weird because i've had this amount of traffic on this server for over 6 months now and the memory usage has NEVER been so high...even on higher loads.

i don't think is relevant but i'm on RHEL4

a day or two ago, the server started crashing....i've been monitoring it since and there is a very high memory usage.

the server has 8gigs of ram.

right now apache is handling 1,428 requests and the server is using 75% of the ram!!!

before, i've hit a load of up to 3,500 requests and i've NEVER used 75% of ram...max was like 40% or 50%

i tried playing with the apache config file with no luck (changing values from very high to very low to defauly, turning things on & off..nothing worked)

can you guys think of anything that might have caused this to happen?

I'd appreciate any help.
thx in advance.

 

[RhythmAce]

If you didn't mess with apache directly, I'm betting you did the monkey thing with something that allocates memory for the processes on your server. Do you have a manager of any type that could be doing this. It would be very odd for a process to just start doing this all on its own.

 

[CGI101]

i'm not even sure what you mean so i certainly did not do it.

can you name some examples of things that allocate memory for the processes on the server?

p.s. i don't know if its relevant but i have cpanel installed.

 

[mbrooks]

can you name some examples of things that allocate memory for the processes on the server?

1. Badly written server-side scripts.

M. Brooks

http://mbrooks.info

 

[RhythmAce]

When you say your server is using 75% ram, do you mean apache or your machine in general. Did you run top or something to find out what process or processes are actually using all that memory? If it's showing apache (httpd) then it could be as mbrooks suggested which is a bad script. Poorly written scripts can create a situation where you have runaway child processes or could put a strain on your system's resources.

 

[CGI101]

well, take a look here....it shows the greatest memory users...

btw, 75% is total system usage.

http://www.megashare.com/images/15315

....

about a week or two ago i remember installing cacti which uses mysql, then a few days later, i found out that there was an exploit out for it which let the user via mysql injection be able to run shell commands on the server....a few minutes later i saw that someone was logged in with cacti's username and password into mysql so i immediately closed cacti's mysql account and removed the actual scripts from the server.

now i don't know what that person aka "hacker" did, so i have no idea what changes he has made...i'm not even sure if he made any changes but thats the last thing i remember happening before the server went like this.

btw, i checked for trojans w/ rkhunter and the only vonurable path it returned was "/etc/.java" which is a folder with 2 0-byte files in it.

....

i just tried this...

httpd stop
service mysql stop
httpd start
(note that i didn't turn on mysql)

i got to 1,500 connections and ram usage was around 10% which is where it should be, so it seems like it has something to do with mysql.

i tried re-creating indexes and that didn't help and now i've ran out of ideas.

any suggestions?

 

[RhythmAce]

Your system comes pretty well secure out of the box but you may have opened some doors for bad guys. If that person you saw was a hacker, it is very possible that he has made another way in. He knows that this will just get his foot in the door. The first thing he would have done is make a way he can get back in after you close that door. You need to keep up on the latest security issues and just keep an eye out for something that looks wrong. If you suspect mysql, The first thing I would do is see if he created a user account or made it so mysql can be accessed from another computer. Unless you have a very good reason for doing otherwise, It should only be accessed from a user@localhost. If your root account can still be accessed without a password (default), I'd change that right now. Another thing you might want to check is all your users and groups. As you know your processes all run as a system user. However they do not have a login shell or password. If any of them do, I'd be suspecting some monkey shines. Also take a look from time to time at the processes that are running on your system. You should know what they are and what they do. If you don't need them or you did not start them, shut them down until you know they are legitimate. This should go without saying - any time your system has been compromised, you should change root's password and pay close attention to your logs.

 

[CGI101]

thx for your input....as far as finding that 2nd way...i was un-successful...foudn 2 accounts that were made under the same usernames "root" and another one of my old usernames, both had their host as local.domain.com...........now i don't know wether those were there before but i removed them anyway, everything is still running okay after i removed them so i figured they were not needed....still makes me wonder why they were there tho.

as far as making another ssh account, i'm 99% sure that he didin't...i have iptables setup on an external firewall that will only let one of MY ips access port 22.

also, i searched with rkhunter, and the only thing it told me to check was the "/etc/.java" directory...i checked it and all i found was 2 empty files...but regardless, just to be safe, i renamed the folder.

any more suggestions will be very appreciated.

 

[RhythmAce]

Well you need a root account for mysql. That account has rights that the other don't. It can also grant privileges to other users. What I was saying is that mysql comes with a root account with no password. You need to make sure this account has a password. It should also be confined to localhost. The exception to this would be if you share databases across a network. In that case you can use a subnet address or domain name. Under no circumstance should use put "Any" in the box. It's hard to keep track of every posible vulnerability on your system but what I consider a great help is "logwatch" which creates a sumerized report from all your logs and e-mails it every morning. It lets me know who and what has been accessing my servers and how often. For example When I see a lot of ftp activity and the logins fail several times, that means someone is trying to get in via ftp. I can open my ftp log and see all the details. I can go back as far as I need and see if these attacks are from the same place and if so, take action. The same goes for my mail server or any other server on my system. Keep on top of all your servers and keep them uptodate. This does not mean to update them every time a new version comes out but check to see if there are any security issues or updates. You should subscribe to the newsletters from RedHat because this is where you'll get this information first. Try not to do upgrades that are not autorized by RH because doing so could create a security somewhere else. For example updating to the latest and greatest php version could cause mysql to become very unstable. There is a reason RedHat moves very slow when it comes to upgrades and that is because they want to make sure their package as a whole remains secure. There is also a reason fedora and other distros are free and refered to as "bleeding edge".

 

[CGI101]

i checked all the accounts including root for mysql and they ALL have passwords...and everything is now using localhost.

btw, now i'm sure that someone got in because
1. my ssl doesn't work
and
2. my paypal IPN doesn't work.

as far as other services ago, their ports are all closed...only ports open are 443 and 80.

as far as php goes, there were 2 security updates that rhn kept telling me about but it was somehow on my "do-not-do" list so it never automatically did them.

i talked to a few ppl about it and they said don't do it because php upgrades and kernel upgrades can mess everything up

...

The following Packages were marked to be skipped by your configuration:

Name Version Rel Reason
-------------------------------------------------------------------------------
kernel 2.6.9 42.0.3.ELPkg name/pattern
kernel-devel 2.6.9 42.0.3.ELPkg name/pattern
kernel-hugemem-devel 2.6.9 42.0.3.ELPkg name/pattern
kernel-smp 2.6.9 42.0.3.ELPkg name/pattern
kernel-smp-devel 2.6.9 42.0.3.ELPkg name/pattern
php 4.3.9 3.22 Pkg name/pattern
php-ldap 4.3.9 3.22 Pkg name/pattern
php-pear 4.3.9 3.22 Pkg name/pattern

 

[CGI101]

i just ran easy apache....re-built apache and php....the ssl problem has been fixed.

don't know about IPN because i havn't been able to test it yet.

but memory issue is still there.

 

[CGI101]

i just caught this....WOW


*******************************************************
*******************************************************
*******************************************************
top - 01:51:03 up 3 days, 5:41, 1 user, load average: 55.29, 37.00, 16.49
Tasks: 2543 total, 19 running, 2524 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.4% us, 94.3% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.2% hi, 0.0% si
Mem: 8312692k total, 8298652k used, 14040k free, 8828k buffers
Swap: 8421368k total, 1499324k used, 6922044k free, 745904k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20487 nobody 16 0 22156 8784 2900 S 28 0.1 0:00.57 httpd
5276 nobody 16 0 28256 12m 3212 R 24 0.2 0:04.81 httpd
21787 nobody 16 0 28992 10m 2824 R 22 0.1 0:01.34 httpd
4955 nobody 15 0 28280 11m 2848 S 22 0.1 0:04.70 httpd
18904 nobody 17 0 22844 8828 2836 R 21 0.1 0:00.90 httpd
4779 nobody 16 0 22276 8744 2864 S 19 0.1 0:02.64 httpd
74 root 16 0 0 0 0 R 18 0.0 60:36.88 kswapd0
26137 nobody 16 0 28088 13m 3212 S 18 0.2 0:04.31 httpd
4837 nobody 16 0 28088 11m 3300 R 17 0.1 0:11.02 httpd
20599 nobody 16 0 27968 13m 2824 S 17 0.2 0:02.36 httpd
18159 nobody 15 0 28120 14m 3336 S 15 0.2 0:02.97 httpd
22031 nobody 18 0 22068 8800 2904 R 13 0.1 0:00.33 httpd
22544 root 20 0 4780 2376 760 R 13 0.0 0:01.25 top
4560 nobody 16 0 22236 8840 2892 S 10 0.1 0:01.50 httpd
20635 nobody 16 0 28088 11m 2844 R 10 0.1 0:01.92 httpd
20934 nobody 15 0 27968 12m 2776 S 7 0.1 0:00.26 httpd
20933 nobody 17 0 22328 8788 2816 R 6 0.1 0:01.26 httpd
4996 nobody 15 0 28120 13m 3332 S 6 0.2 0:03.19 httpd
2657 mysql 15 0 134m 22m 2156 S 4 0.3 91:41.41 mysqld
6313 nobody 15 0 28088 12m 3216 S 4 0.2 0:03.51 httpd
20445 nobody 15 0 27968 11m 2816 S 3 0.1 0:01.90 httpd
10561 nobody 16 0 22964 9572 3304 S 1 0.1 0:02.38 httpd
21004 nobody 15 0 22844 8936 2820 S 1 0.1 0:00.04 httpd
2839 root 15 0 0 0 0 S 1 0.0 0:19.49 kjournald
4734 root 16 0 232m 2492 1472 S 1 0.0 3:38.10 dsm_sa_datamgr3
4867 nobody 16 0 28560 13m 2872 S 1 0.2 0:02.31 httpd
5245 nobody 15 0 22220 9176 3244 S 1 0.1 0:02.95 httpd
5920 nobody 15 0 22220 9232 3236 S 1 0.1 0:02.52 httpd
7384 nobody 15 0 22448 9296 3212 R 1 0.1 0:02.44 httpd
7850 nobody 15 0 28088 13m 2844 S 1 0.2 0:02.47 httpd
7899 nobody 15 0 28088 11m 2848 S 1 0.1 0:04.50 httpd
12654 nobody 16 0 22996 9528 3236 S 1 0.1 0:02.60 httpd
14875 nobody 16 0 22276 9248 3240 S 1 0.1 0:02.79 httpd
19701 nobody 15 0 28088 12m 2848 S 1 0.2 0:05.18 httpd
20702 nobody 16 0 22964 9432 3308 S 1 0.1 0:04.01 httpd
20742 nobody 15 0 22188 8776 2844 S 1 0.1 0:02.86 httpd
30513 nobody 16 0 22308 9260 3236 S 1 0.1 0:01.94 httpd
12111 nobody 15 0 22188 8864 2840 S 1 0.1 0:01.08 httpd
24346 nobody 16 0 23028 9448 3216 S 1 0.1 0:02.29 httpd
*******************************************************
*******************************************************
*******************************************************

i couldn't barely ssh into the machine...looks like i caught right before it was about to crash...i just immediately stopped apache.

any ideas?

 

[CGI101]

i re-installed mysql, issue is still there. 8(

 

[CGI101]

i don't friggin believe this.

took me 2 weeks to figure out ONE line of code that was causing this issue.

all downloads were being forced to be transferred at the speed of 5megs/sec.

i'm soooo dumb.

 

[RhythmAce]

Imagine how badly I feel. I should have asked you, hey are you forcing data to be tranfered at 5 megs/sec?

 

[CGI101]

with fread.