linux hangs running sendmail

[rejohn]

I'm running an IBM Xseries M8685 server with 8-cpu's and 8gb of memory, the OS is Redhat Linux AS 3. I'm using quotas and almost every night the server hangs. If i press Alt+SysRq+W to walk through the processors, it returns sendmail and sometimes swapper as the processes running before the freeze. Has anyone else had this problem?

 

[mrballcb]

No, not sendmail, it's usually apache for us. I'd say you should put together a little perl script that daemonizes and does a 'ps auxwww' and/or 'uptime' and/or 'free' every 5 seconds or so, write it to a file someplace that won't get deleted when you reboot, and try to catch the series of events that are causing it to hang. It's a hit or miss thing.

The biggest red flag I see is that cron jobs run at night. Make sure that:
a) root is exempted from quotas, or has a pretty high setting.
b) the sendmail user has a pretty high setting too.

Take a look at all user cron jobs (look in /var/spool/cron/) to see if anything is running that you don't know about. But I can say that if some user cron job is crashing your machine, you have bigger problems because that's just something that should not happen.

 

[rejohn]

if root is to exempted from quotas, what about users who have sudo privs???

 

[mrballcb]

I should have used a different word than "exempted". A better word would be "quite a bit larger than is currently used". Remember that simple things like log rotations can create very large temp files, most of which are owned by root. If enough things are running at the same time, you might find that the quota is being hit. But I think that's a shot in the dark.

I still would like to see the series of events and exact time that it's hanging. Nothing in the log files? Is it an out of memory condition? Or is it a hard hang? It can't be too hard of a hang if you can use the SysReq keys to walk through the processes.

 

[rejohn]

her is a list or processes 1 minute before the server hanged:
-------------------------------------------------------------
uptime...................
14:30:00 up 6:20, 1 user, load average: 9.63, 10.36, 9.84 memory usage.............
total used free shared buffers cached
Mem: 8047296 7999024 48272 0 42700 7509832
-/+ buffers/cache: 446492 7600804
Swap: 2096220 0 2096220

USER is running:.....
1 instance(s) of >> STAT START TIME COMMAND

apache is running:.....
1 instance(s) of >> S 14:00 1:09 /usr/sbin/httpd
1 instance(s) of >> S 14:02 0:52 /usr/sbin/httpd
1 instance(s) of >> S 14:02 1:05 /usr/sbin/httpd
1 instance(s) of >> S 14:04 0:57 /usr/sbin/httpd
1 instance(s) of >> S 14:04 0:58 /usr/sbin/httpd
1 instance(s) of >> S 14:04 0:56 /usr/sbin/httpd
1 instance(s) of >> S 14:09 0:43 /usr/sbin/httpd
1 instance(s) of >> S 14:09 0:42 /usr/sbin/httpd
1 instance(s) of >> S 14:09 0:37 /usr/sbin/httpd
1 instance(s) of >> S 14:11 0:37 /usr/sbin/httpd
1 instance(s) of >> S 14:12 0:36 /usr/sbin/httpd
1 instance(s) of >> S 14:13 0:30 /usr/sbin/httpd
1 instance(s) of >> S 14:13 0:32 /usr/sbin/httpd
1 instance(s) of >> S 14:15 0:26 /usr/sbin/httpd
1 instance(s) of >> S 14:16 0:22 /usr/sbin/httpd
1 instance(s) of >> S 14:18 0:30 /usr/sbin/httpd
1 instance(s) of >> S 14:18 0:19 /usr/sbin/httpd
1 instance(s) of >> S 14:18 0:24 /usr/sbin/httpd
1 instance(s) of >> S 14:20 0:15 /usr/sbin/httpd
1 instance(s) of >> S 14:20 0:17 /usr/sbin/httpd

daemon is running:.....
1 instance(s) of >> S 08:13 0:00 /usr/sbin/atd

mailman is running:.....
1 instance(s) of >> D 08:13 2:28 /usr/bin/python /var/mailman/bin
1 instance(s) of >> D 14:29 0:00 /usr/bin/python -S /var/mailman/
1 instance(s) of >> S 08:13 0:00 /usr/bin/python /var/mailman/bin
2 instance(s) of >> S 08:13 0:54 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S 08:13 0:53 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S 08:13 0:55 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S 08:13 2:10 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S 08:13 1:17 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S 08:13 0:00 /usr/bin/python /var/mailman/bin

mysql is running:.....
1 instance(s) of >> S 08:12 0:22 /usr/libexec/mysqld --defaults-f

ntp is running:.....
1 instance(s) of >> SL 08:12 0:01 ntpd -U ntp -p /var/run/ntpd.pi

parker is running:.....
1 instance(s) of >> S 14:05 0:00 imapd
1 instance(s) of >> S 13:55 0:00 imapd

rejohn is running:.....
1 instance(s) of >> S 08:13 0:02 sshd: rejohn@pts/0
1 instance(s) of >> S 08:13 0:00 -bash

root is running:.....
1 instance(s) of >> D 11:12 19:26 sendmail: ./jARNQE5V014141 from
1 instance(s) of >> D 14:29 0:00 sendmail: jATKTstg006001 mcvs2.o
1 instance(s) of >> D 14:29 0:00 /usr/bin/perl -w /usr/bin/mrtg /
1 instance(s) of >> D 14:30 0:00 sendmail: ./jATKTste006001 from
1 instance(s) of >> D 14:30 0:00 procmail -f MAILER-DAEMON@mail.u
1 instance(s) of >> D 14:30 0:00 sendmail: ./jATKTstg006001 from
1 instance(s) of >> D 09:12 52:22 sendmail: ./jARBOnRW018391 from
1 instance(s) of >> D 13:12 8:46 sendmail: ./jASN7gTm020162 from
1 instance(s) of >> D 14:12 0:52 sendmail: ./jATHbPHH002495 from
1 instance(s) of >> DW 08:11 15:56 [kjournald]
1 instance(s) of >> R 08:12 55:17 sendmail: ./jARBOnRV018391 from
1 instance(s) of >> R 14:29 0:00 ipop3d
1 instance(s) of >> R 14:29 0:00 sendmail: jATKU0ZV006093 mcvs2.o
1 instance(s) of >> R 14:29 0:00 /usr/bin/perl /usr/sysadm/bin/sc
1 instance(s) of >> R 14:29 0:00 /usr/bin/getent passwd
1 instance(s) of >> R 14:29 0:00 ipop3d
1 instance(s) of >> R 14:30 0:00 /bin/ps auxwww
1 instance(s) of >> R 14:30 0:00 ssh testaccount@ocean.otr.usm.ed
1 instance(s) of >> R 14:30 0:00 ipop3d
1 instance(s) of >> R 14:30 0:00 sendmail: ./jATKU0ZV006093 from
1 instance(s) of >> R 14:30 0:00 ipop3d
1 instance(s) of >> S 08:09 0:22 init
1 instance(s) of >> S 08:57 0:00 ipop3d
1 instance(s) of >> S 10:12 28:54 sendmail: ./jAQJ4fOW002468 from
1 instance(s) of >> S 08:12 1:00 syslogd -m 0
1 instance(s) of >> S 08:12 0:00 klogd -x
1 instance(s) of >> S 08:12 0:09 irqbalance
1 instance(s) of >> S 08:12 0:00 /usr/sbin/sshd
1 instance(s) of >> S 08:12 0:00 /bin/sh /usr/bin/safe_mysqld --d
1 instance(s) of >> S 08:12 1:36 sendmail: accepting connections
1 instance(s) of >> S 08:12 0:35 /opt/tivoli/tsm/client/ba/bin/ds
1 instance(s) of >> S 08:12 0:00 gpm -t imps2 -m /dev/mouse
1 instance(s) of >> S 08:13 0:02 /usr/sbin/httpd
1 instance(s) of >> S 08:13 0:00 /usr/sbin/radiusd -y
1 instance(s) of >> S 08:13 0:01 crond
1 instance(s) of >> S 08:13 0:00 rhnsd --interval 240
1 instance(s) of >> S 08:13 0:00 sshd: rejohn [priv]
1 instance(s) of >> S 14:25 0:00 sendmail: server mcvs1.otr.usm.e
1 instance(s) of >> S 14:27 0:00 sendmail: server mcvs3.otr.usm.e
1 instance(s) of >> S 09:01 0:00 ipop3d
1 instance(s) of >> S 14:28 0:00 sendmail: server mcvs1.otr.usm.e
1 instance(s) of >> S 08:16 5:18 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 14:28 0:00 sendmail: server mcvs1.otr.usm.e
1 instance(s) of >> S 14:29 0:00 sendmail: server mcvs1.otr.usm.e
1 instance(s) of >> S 14:29 0:00 sendmail: server mcvs3.otr.usm.e
27 instance(s) of >> S 14:29 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 09:43 0:00 ipop3d
5 instance(s) of >> S 14:29 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 14:29 0:00 sendmail: server mcvs3.otr.usm.e
5 instance(s) of >> S 14:29 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 14:29 0:00 sendmail: ./jATKToeA005941 smtp.
8 instance(s) of >> S 14:29 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 14:29 0:00 imapd
1 instance(s) of >> S 14:29 0:00 sendmail: ./jATKTuvJ006044 smtp.
4 instance(s) of >> S 14:29 0:00 xinetd -stayalive -pidfile /var/
2 instance(s) of >> S 14:29 0:00 procmail -f MAILER-DAEMON@mail.u
3 instance(s) of >> S 14:29 0:00 CROND
1 instance(s) of >> S 14:29 0:00 /bin/ksh /ocean/sysadm/rejohn/pr
2 instance(s) of >> S 14:29 0:00 CROND
1 instance(s) of >> S 14:29 0:00 /usr/bin/perl /root/shadow_copy.
1 instance(s) of >> S 14:29 0:00 /bin/sh -c /usr/sysadm/bin/scamp
1 instance(s) of >> S 14:29 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 14:30 0:00 sendmail: ./jATKU0ZT006093 from
1 instance(s) of >> S 14:30 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 14:30 0:00 /bin/sort +0 -1 +7 -8
1 instance(s) of >> S 14:30 0:00 sh -c ssh testaccount@ocean.otr.
1 instance(s) of >> S 14:30 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 14:30 0:00 procmail -f confirm@facebook.com
1 instance(s) of >> S 09:05 0:00 ipop3d
1 instance(s) of >> S 12:12 9:20 sendmail: ./jASNi8fU003381 from
1 instance(s) of >> S 08:30 0:00 ipop3d
1 instance(s) of >> S 08:33 0:00 ipop3d
1 instance(s) of >> S 13:42 0:00 ipop3d
1 instance(s) of >> S 13:14 0:00 ipop3d
1 instance(s) of >> S 14:10 0:00 ipop3d
1 instance(s) of >> S 09:22 0:00 ipop3d
1 instance(s) of >> S 14:13 0:00 ipop3d
1 instance(s) of >> S 11:28 0:00 ipop3d
1 instance(s) of >> S 10:06 0:00 ipop3d
1 instance(s) of >> S 08:51 0:00 ipop3d
1 instance(s) of >> S 14:23 0:00 ipop3d
6 instance(s) of >> S< 08:13 0:00 ncftpd -d /usr/local/etc/ncftpd
1 instance(s) of >> SN 08:13 0:00 ncftpd -d /usr/local/etc/ncftpd
1 instance(s) of >> SW 08:09 0:00 [migration/0]
1 instance(s) of >> SW 08:09 0:00 [migration/1]
1 instance(s) of >> SW 08:09 0:00 [migration/2]
1 instance(s) of >> SW 08:09 0:00 [migration/3]
1 instance(s) of >> SW 08:09 0:00 [migration/4]
1 instance(s) of >> SW 08:09 0:00 [migration/5]
1 instance(s) of >> SW 08:09 0:00 [migration/6]
1 instance(s) of >> SW 08:09 0:00 [migration/7]
1 instance(s) of >> SW 08:09 0:00 [keventd]
1 instance(s) of >> SW 08:09 0:12 [kswapd]
1 instance(s) of >> SW 08:09 0:08 [kscand]
1 instance(s) of >> SW 08:09 0:00 [bdflush]
1 instance(s) of >> SW 08:09 0:12 [kupdated]
1 instance(s) of >> SW 08:09 0:00 [mdrecoveryd]
1 instance(s) of >> SW 08:09 0:00 [ahc_dv_0]
1 instance(s) of >> SW 08:09 0:00 [ahc_dv_1]
1 instance(s) of >> SW 08:09 0:00 [scsi_eh_0]
1 instance(s) of >> SW 08:09 0:00 [scsi_eh_1]
1 instance(s) of >> SW 08:10 0:00 [scsi_eh_2]
1 instance(s) of >> SW 08:10 0:10 [kjournald]
1 instance(s) of >> SW 08:10 0:00 [khubd]
1 instance(s) of >> SW 08:11 0:00 [kjournald]
1 instance(s) of >> SW 08:11 3:03 [kjournald]
1 instance(s) of >> SW 08:11 0:09 [kjournald]
1 instance(s) of >> SW 08:11 0:19 [kjournald]
1 instance(s) of >> SWN 08:09 0:00 [ksoftirqd/0]
1 instance(s) of >> SWN 08:09 0:00 [ksoftirqd/1]
1 instance(s) of >> SWN 08:09 0:00 [ksoftirqd/2]
1 instance(s) of >> SWN 08:09 0:00 [ksoftirqd/3]
1 instance(s) of >> SWN 08:09 0:00 [ksoftirqd/4]
1 instance(s) of >> SWN 08:09 0:00 [ksoftirqd/5]
1 instance(s) of >> SWN 08:09 0:00 [ksoftirqd/6]
1 instance(s) of >> SWN 08:09 0:00 [ksoftirqd/7]
1 instance(s) of >> S 14:29 0:00 procmail -f MAILER-DAEMON@mail.u
1 instance(s) of >> S 08:13 0:00 /sbin/mingetty tty2
1 instance(s) of >> S 08:13 0:00 /sbin/mingetty tty3
1 instance(s) of >> S 08:13 0:00 /sbin/mingetty tty4
1 instance(s) of >> S 08:13 0:00 /sbin/mingetty tty5
1 instance(s) of >> S 08:13 0:00 /sbin/mingetty tty6
1 instance(s) of >> S 10:25 0:00 /sbin/mingetty tty1
1 instance(s) of >> S 08:13 0:00 su -
1 instance(s) of >> S 08:13 0:01 -bash

smmsp is running:.....
1 instance(s) of >> S 08:12 0:00 sendmail: Queue runner@01:00:00
1 instance(s) of >> S 14:29 0:00 /usr/sbin/sendmail -FCronDaemon

w123721 is running:.....
1 instance(s) of >> R 14:30 0:00 procmail -f vgyhjn@emergent.com

w124825 is running:.....
1 instance(s) of >> S 08:12 0:10 imapd

w143766 is running:.....
1 instance(s) of >> R 14:30 0:00 ipop3d

w300296 is running:.....
1 instance(s) of >> S 08:42 0:00 imapd

w302291 is running:.....
1 instance(s) of >> S 14:28 0:00 ipop3d

w302651 is running:.....
1 instance(s) of >> S 08:54 0:01 imapd

w304303 is running:.....
1 instance(s) of >> S 08:14 0:02 imapd
1 instance(s) of >> S 08:15 0:01 imapd

w304506 is running:.....
1 instance(s) of >> D 14:30 0:00 procmail -f Haband@eway10.com -t

w304905 is running:.....
1 instance(s) of >> S 13:29 0:02 imapd

w305787 is running:.....
1 instance(s) of >> D 14:30 0:00 ipop3d

w306005 is running:.....
1 instance(s) of >> S 13:57 0:00 imapd

w306038 is running:.....
1 instance(s) of >> S 08:35 0:02 imapd

w306888 is running:.....
1 instance(s) of >> D 14:30 0:00 procmail -f bnc-88ec57eb-ccc1-4f

w306906 is running:.....
1 instance(s) of >> S 14:07 0:01 imapd

w430584 is running:.....
1 instance(s) of >> S 14:29 0:00 imapd

w453741 is running:.....
1 instance(s) of >> S 09:11 0:01 imapd

w473890 is running:.....
1 instance(s) of >> R 14:30 0:00 imapd

w487907 is running:.....
1 instance(s) of >> S 14:29 0:00 ipop3d

w492038 is running:.....
1 instance(s) of >> S 08:20 0:01 imapd

w499861 is running:.....
1 instance(s) of >> R 14:29 0:00 imapd

w513235 is running:.....
1 instance(s) of >> R 14:30 0:00 imapd

w535313 is running:.....
1 instance(s) of >> S 14:29 0:00 ipop3d

w535653 is running:.....
1 instance(s) of >> R 14:30 0:00 ipop3d

w536460 is running:.....
netstat.............
tcp 102 0 131.95.82.42:995 131.95.240.76:3277 ESTABLISHED
tcp 0 0 131.95.82.42:22 131.95.175.215:4596 ESTABLISHED
tcp 0 0 127.0.0.1:143 127.0.0.1:49184 ESTABLISHED
tcp 0 30 131.95.82.42:995 131.95.240.112:3800 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.121.69:1378 ESTABLISHED
tcp 0 0 127.0.0.1:49139 127.0.0.1:143 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.240.88:1774 ESTABLISHED
tcp 0 0 131.95.82.42:995 4.231.83.219:46324 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1246 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1232 ESTABLISHED
tcp 0 0 127.0.0.1:49184 127.0.0.1:143 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1231 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.240.220:1487 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1268 ESTABLISHED
tcp 102 0 131.95.82.42:995 198.49.215.87:1785 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.108.168:57645 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.235.27:1528 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1249 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1254 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1255 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1296 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1559 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.237.89:3676 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.108.168:57814 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.85.41:2970 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.175.207:1092 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.204.24:2058 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.175.207:1090 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1584 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.229.109:4191 ESTABLISHED
tcp 0 4152 131.95.82.42:995 131.95.164.116:3077 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.175.179:1320 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.165.21:1910 ESTABLISHED
tcp 0 0 131.95.82.42:995 206.51.26.206:38378 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.108.112:3259 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.203.103:3639 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.104.16:1772 ESTABLISHED
tcp 0 0 127.0.0.1:143 127.0.0.1:49139 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.100.136:3734 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.55:45087 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.55:45333 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.198.11:1669 ESTABLISHED
tcp 0 146 131.95.82.42:443 68.63.224.89:4254 ESTABLISHED
tcp 0 146 131.95.82.42:443 68.63.224.89:4255 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.203.26:2794 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.55:45398 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.211:4049 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.237.80:1189 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.240.81:2477 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.108.201:1368 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.192.161:1183 ESTABLISHED
tcp 0 1735 131.95.82.42:995 131.95.172.61:1636 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.240.13:2065 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.170.123:1841 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.12.41:1472 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.219:2858 ESTABLISHED
tcp 0 1735 131.95.82.42:995 68.159.217.224:50636 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.22.11:1701 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.150.220:3274 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.56:42555 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.171.29:2275 ESTABLISHED
tcp 120 0 131.95.82.42:995 68.222.73.183:50126 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.240.78:3717 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.58.40:2092 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.56:42585 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.17.63:3849 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.237.71:3470 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.29.15:1039 ESTABLISHED
tcp 0 161 131.95.82.42:49192 131.95.82.112:1521 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.50.148:1097 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.210.16:2759 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.184.74:3076 ESTABLISHED
tcp 0 0 131.95.82.42:443 131.95.92.38:2961 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.59.229:1222 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.56:42636 ESTABLISHED
tcp 0 1735 131.95.82.42:995 131.95.218.88:2906 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.17.67:54405 ESTABLISHED
tcp 0 0 131.95.82.42:47543 131.95.82.56:25 ESTABLISHED
tcp 0 0 131.95.82.42:49097 131.95.82.56:25 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.74.24:2462 ESTABLISHED
tcp 0 0 131.95.82.42:46753 131.95.82.56:25 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.213.51:2653 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.242.20:1114 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.237.183:1538 ESTABLISHED
tcp 78 0 131.95.82.42:995 131.95.204.162:3849 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.56:42747 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.56:42748 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.173.219:1792 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.56:42215 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.20.18:1152 ESTABLISHED
tcp 0 0 131.95.82.42:993 129.173.105.114:52375 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.101:1493 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.172.91:2715 ESTABLISHED
tcp 120 0 131.95.82.42:995 131.95.85.166:4719 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.165.170:1680 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.37:1699 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.230.13:4028 ESTABLISHED
tcp 78 0 131.95.82.42:995 131.95.12.55:2419 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.191.20:1510 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.109:3575 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.197.122:1408 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.113.178:1053 ESTABLISHED
tcp 35 0 131.95.82.42:995 131.95.211.12:3391 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.85.50:1408 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.213.38:2336 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.170.113:1391 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.115.204:43764 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.100.99:1708 ESTABLISHED
tcp 0 6375 131.95.82.42:995 4.253.132.111:1516 ESTABLISHED
tcp 43 0 131.95.82.42:995 131.95.235.32:1622 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.149.62:1593 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.164.215:4870 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.57:44257 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.172.171:2420 ESTABLISHED
udp 0 0 131.95.82.42:41168 131.95.84.2:53 ESTABLISHED
udp 0 0 131.95.82.42:41176 198.49.215.21:53 ESTABLISHED
udp 0 0 131.95.82.42:41184 131.95.242.10:53 ESTABLISHED

End of program: proclist

 

[mrballcb]

Lots of imap connections, but I would assume they're idle. That seems ok to me.

You have plenty of RAM, and most of it is being used for buffers. That seems ok to me.

The load of your system is hanging around 10, is that normal for daytime too, or is that happening only at night?

If this always happens at the same time every night, I'd suggest killing sendmail in a cron job about 2 minutes before it hangs. Then start it up about 2 minutes after it hangs. Then see if it has anything to do with it.

If it doesn't always happen, then change the frequency of the above output from once per minute to 6 times per minute to try and get more detailed system happenings.

Just as an experiment, see what happens if you turn off swap. It's not using any of it in normal operation, so in theory it should make zero difference. But I've found that if something (such as apache in our case) starts wanting lots of memory, the kernel tries to give it and starts swapping like mad, then killing processes when it runs out of swap, leaving you with a locked up system. But it usually says that the kernel was killing processes on the console. If yours is not doing this, then you have a different problem than what I'm used to seeing.

 

[rejohn]

I've turn off swap and it made no difference, I also increased the frequency of the process list script to every 30 seconds this is what we caught before our last hang. I notice we're having an I/O bottleneck.
---------------------------------------------------------------

top ...............


09:59:02 up 12:07, 0 users, load average: 23.27, 27.45, 27.82
246 processes: 244 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 12.7% 0.0% 1.3% 0.0% 0.2% 85.6% 0.0%
cpu00 13.3% 0.0% 1.7% 0.0% 0.8% 83.9% 0.0%
cpu01 0.0% 0.0% 0.0% 0.0% 0.0% 100.0% 0.0%
cpu02 18.7% 0.0% 0.8% 0.0% 0.8% 79.4% 0.0%
cpu03 23.2% 0.0% 4.4% 0.0% 0.0% 72.3% 0.0%
cpu04 26.7% 0.0% 2.6% 0.0% 0.0% 70.5% 0.0%
cpu05 0.0% 0.0% 0.0% 0.0% 0.0% 100.0% 0.0%
cpu06 18.0% 0.0% 0.0% 0.0% 0.0% 81.9% 0.0%
cpu07 1.8% 0.0% 0.9% 0.0% 0.0% 97.2% 0.0%
Mem: 8047296k av, 8024536k used, 22760k free, 0k shrd, 68904k buff
2619876k actv, 3912984k in_d, 128820k in_c
Swap: 2096220k av, 0k used, 2096220k free 7325648k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
11941 w137302 16 0 2104 2104 1764 D 3.2 0.0 0:00 4 ipop3d
11947 w300958 16 0 2104 2104 1764 D 2.2 0.0 0:00 4 ipop3d
12343 w301176 24 0 2088 2088 1752 D 2.2 0.0 0:00 6 ipop3d
12344 w306777 24 0 2084 2084 1752 D 2.0 0.0 0:00 3 ipop3d
12681 apache 16 0 17188 16M 8328 R 1.4 0.2 1:28 2 httpd
12332 root 24 0 1144 1144 768 R 1.0 0.0 0:00 7 top
945 root 15 0 0 0 0 DW 0.1 0.0 59:29 5 kjournald
1376 root 22 0 920 920 776 S 0.1 0.0 5:00 5 xinetd
1 root 15 0 500 500 436 S 0.0 0.0 1:13 7 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
4 root RT 0 0 0 0 SW 0.0 0.0 0:00 2 migration/2
5 root RT 0 0 0 0 SW 0.0 0.0 0:00 3 migration/3
6 root RT 0 0 0 0 SW 0.0 0.0 0:00 4 migration/4
7 root RT 0 0 0 0 SW 0.0 0.0 0:00 5 migration/5
8 root RT 0 0 0 0 SW 0.0 0.0 0:00 6 migration/6
9 root RT 0 0 0 0 SW 0.0 0.0 0:00 7 migration/7
10 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
11 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
12 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
13 root 39 19 0 0 0 SWN 0.0 0.0 0:00 2 ksoftirqd/2
14 root 34 19 0 0 0 SWN 0.0 0.0 0:00 3 ksoftirqd/3
15 root 34 19 0 0 0 SWN 0.0 0.0 0:00 4 ksoftirqd/4
16 root 34 19 0 0 0 SWN 0.0 0.0 0:00 5 ksoftirqd/5
17 root 34 19 0 0 0 SWN 0.0 0.0 0:00 6 ksoftirqd/6
18 root 34 19 0 0 0 SWN 0.0 0.0 0:00 7 ksoftirqd/7
21 root 15 0 0 0 0 SW 0.0 0.0 0:00 6 bdflush
19 root 15 0 0 0 0 SW 0.0 0.0 0:36 7 kswapd
20 root 15 0 0 0 0 SW 0.0 0.0 0:12 1 kscand
22 root 15 0 0 0 0 SW 0.0 0.0 0:31 7 kupdated
23 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
30 root 15 0 0 0 0 SW 0.0 0.0 0:00 4 ahc_dv_0
31 root 15 0 0 0 0 SW 0.0 0.0 0:00 5 ahc_dv_1
32 root 25 0 0 0 0 SW 0.0 0.0 0:00 4 scsi_eh_0
33 root 25 0 0 0 0 SW 0.0 0.0 0:00 4 scsi_eh_1
35 root 21 0 0 0 0 SW 0.0 0.0 0:00 1 scsi_eh_2
38 root 15 0 0 0 0 SW 0.0 0.0 0:30 0 kjournald
94 root 25 0 0 0 0 SW 0.0 0.0 0:00 7 khubd
944 root 15 0 0 0 0 SW 0.0 0.0 0:00 7 kjournald
946 root 15 0 0 0 0 DW 0.0 0.0 3:45 3 kjournald
947 root 15 0 0 0 0 SW 0.0 0.0 0:28 4 kjournald
948 root 15 0 0 0 0 SW 0.0 0.0 0:46 5 kjournald
1262 root 15 0 716 716 596 S 0.0 0.0 1:23 1 syslogd
1267 root 15 0 460 460 396 S 0.0 0.0 0:01 5 klogd
1278 root 15 0 464 464 396 S 0.0 0.0 0:31 3 irqbalance
1364 root 25 0 1484 1484 1236 S 0.0 0.0 0:00 7 sshd
1389 ntp 15 0 2576 2576 2200 S 0.0 0.0 0:03 7 ntpd
1402 root 25 0 1168 1168 1020 S 0.0 0.0 0:00 1 safe_mysqld
1429 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 3 mysqld
1431 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 4 mysqld
1464 root 16 0 2528 2528 1832 S 0.0 0.0 3:54 7 sendmail
1465 root 15 0 8192 8188 2320 D 0.0 0.1 91:02 3 sendmail
1474 smmsp 15 0 2224 2216 1668 S 0.0 0.0 0:00 0 sendmail
1483 root 15 0 3440 3440 2372 S 0.0 0.0 1:44 6 dsmc
1493 root 25 0 3440 3440 2372 S 0.0 0.0 0:00 6 dsmc
1503 root 25 0 472 472 412 S 0.0 0.0 0:00 2 gpm
1532 root 15 0 9400 9400 6572 S 0.0 0.1 0:03 5 httpd
1542 root 12 -3 1996 1996 924 S < 0.0 0.0 0:01 1 ncftpd
1543 root 16 1 972 972 840 S N 0.0 0.0 0:02 5 ncftpd
1552 root 13 -3 2060 2060 980 S < 0.0 0.0 0:00 1 ncftpd
1553 root 12 -3 2060 2060 980 S < 0.0 0.0 0:00 1 ncftpd
1556 root 22 -3 1972 1972 900 S < 0.0 0.0 0:00 1 ncftpd
1560 root 15 0 3264 3264 1384 S 0.0 0.0 0:00 7 radiusd
1561 root 16 0 3264 3264 1384 S 0.0 0.0 0:00 7 radiusd
1562 root 15 0 3264 3264 1384 S 0.0 0.0 0:00 1 radiusd
1563 root 25 0 3264 3264 1384 S 0.0 0.0 0:00 7 radiusd
1564 root 16 0 3264 3264 1384 S 0.0 0.0 0:00 6 radiusd
1565 root 15 0 3264 3264 1384 S 0.0 0.0 0:00 7 radiusd
1578 root 15 0 640 640 552 S 0.0 0.0 0:04 7 crond
1598 daemon 15 0 580 580 500 S 0.0 0.0 0:02 7 atd
1610 root 15 0 576 576 512 S 0.0 0.0 0:00 1 rhnsd
1640 mailman 25 0 4652 4652 1880 S 0.0 0.0 0:00 4 python
1647 mailman 15 0 4740 4740 1904 S 0.0 0.0 2:49 1 python
1648 mailman 15 0 4676 4676 1856 S 0.0 0.0 2:51 5 python
1649 mailman 15 0 5336 5336 1948 S 0.0 0.0 2:50 4 python
1650 mailman 15 0 10140 9.9M 1972 S 0.0 0.1 5:28 2 python
1651 mailman 15 0 4708 4708 1856 S 0.0 0.0 2:51 2 python
1653 mailman 15 0 7876 7876 1976 S 0.0 0.0 4:29 2 python
1657 mailman 15 0 7756 7756 1936 S 0.0 0.0 3:23 0 python
1658 mailman 15 0 4664 4664 1856 S 0.0 0.0 0:00 1 python
1659 root 23 0 432 432 372 S 0.0 0.0 0:00 0 mingetty
1660 root 23 0 428 428 372 S 0.0 0.0 0:00 1 mingetty
1661 root 23 0 424 424 372 S 0.0 0.0 0:00 6 mingetty
1662 root 23 0 424 424 372 S 0.0 0.0 0:00 0 mingetty
1663 root 23 0 424 424 372 S 0.0 0.0 0:00 0 mingetty
1664 root 23 0 424 424 372 S 0.0 0.0 0:00 1 mingetty
2819 w124825 15 0 2304 2304 1912 S 0.0 0.0 0:27 3 imapd
6275 root 15 0 1968 1968 1640 S 0.0 0.0 0:00 0 ipop3d
16293 root 15 0 1964 1964 1640 S 0.0 0.0 0:00 6 ipop3d
27708 root 15 0 8032 8028 2352 S 0.0 0.0 85:02 1 sendmail
25531 root 16 0 8144 8140 2352 D 0.0 0.1 76:39 4 sendmail
32228 root 18 0 8040 8036 2352 S 0.0 0.0 61:43 7 sendmail
7927 root 15 0 7908 7904 2352 D 0.0 0.0 49:16 1 sendmail
14007 root 15 0 7852 7848 2352 S 0.0 0.0 43:24 5 sendmail
29540 root 15 0 7700 7696 2352 S 0.0 0.0 36:31 2 sendmail
13782 root 15 0 7688 7684 2352 S 0.0 0.0 34:10 6 sendmail
25330 root 21 0 7592 7588 2352 S 0.0 0.0 21:42 1 sendmail
6063 root 18 0 7488 7484 2352 D 0.0 0.0 14:50 2 sendmail
4382 root 20 0 7356 7352 2352 D 0.0 0.0 11:40 5 sendmail
19555 w300296 15 0 2892 2892 1932 S 0.0 0.0 0:01 4 imapd
15987 w492038 15 0 2576 2576 1948 S 0.0 0.0 0:01 1 imapd
27821 w306038 15 0 3180 3180 1952 S 0.0 0.0 0:02 1 imapd
5242 root 15 0 1892 1892 1576 S 0.0 0.0 0:00 0 ipop3d
8063 w306038 15 0 2924 2924 1932 S 0.0 0.0 0:01 5 imapd
10394 root 20 0 6896 6892 2352 S 0.0 0.0 5:24 6 sendmail
25646 apache 15 0 16136 15M 8672 S 0.0 0.2 1:31 7 httpd
26468 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 2 mysqld
31494 apache 15 0 14972 14M 8372 S 0.0 0.1 0:37 2 httpd
31605 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 5 mysqld
12641 apache 15 0 22108 21M 8328 S 0.0 0.2 1:21 7 httpd
12646 apache 15 0 15344 14M 8604 S 0.0 0.1 1:16 5 httpd
12647 apache 15 0 18312 17M 8440 S 0.0 0.2 1:04 7 httpd
12729 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 3 mysqld
12736 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 7 mysqld
13384 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 7 mysqld
13542 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 7 mysqld
14381 apache 16 0 19352 18M 8344 S 0.0 0.2 1:18 2 httpd
14547 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 7 mysqld
20318 apache 16 0 14976 14M 8636 S 0.0 0.1 0:58 1 httpd
20576 apache 15 0 15940 15M 8440 S 0.0 0.1 0:59 4 httpd
20631 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 1 mysqld
21126 apache 16 0 14976 14M 8324 S 0.0 0.1 1:01 0 httpd
21276 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 1 mysqld
24175 w487907 15 0 3548 3548 1876 S 0.0 0.0 0:03 3 ipop3d
24264 root 18 0 3204 3204 2308 S 0.0 0.0 0:00 1 sendmail
25323 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 3 mysqld
27608 w302651 15 0 3568 3568 1948 S 0.0 0.0 0:02 5 imapd
28700 w303825 15 0 2392 2392 1916 S 0.0 0.0 0:01 5 imapd
29054 apache 17 0 14968 14M 8328 S 0.0 0.1 0:40 1 httpd
29059 apache 16 0 14684 14M 8376 S 0.0 0.1 0:35 0 httpd
29060 apache 15 0 14380 14M 8320 S 0.0 0.1 0:41 3 httpd
29152 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 6 mysqld
30507 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 0 mysqld
30803 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 7 mysqld
30988 apache 15 0 14728 14M 8336 S 0.0 0.1 0:35 2 httpd
31187 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 1 mysqld
32385 apache 15 0 14640 14M 8324 S 0.0 0.1 0:33 3 httpd
32414 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 6 mysqld
32419 apache 16 0 15304 14M 8372 S 0.0 0.1 0:32 0 httpd
32420 apache 15 0 14908 14M 8336 S 0.0 0.1 0:30 1 httpd
1386 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 2 mysqld
1593 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 3 mysqld
1826 w105392 15 0 2284 2284 1912 S 0.0 0.0 0:00 0 imapd
6591 root 16 0 1896 1896 1576 S 0.0 0.0 0:00 5 ipop3d
8189 root 15 0 3868 3868 1844 D 0.0 0.0 0:41 7 sendmail
8371 root 12 -3 1996 1996 924 S < 0.0 0.0 0:00 1 ncftpd
8406 root 12 -3 1996 1996 924 S < 0.0 0.0 0:00 1 ncftpd
9626 apache 15 0 15056 14M 8136 S 0.0 0.1 0:03 1 httpd
9657 apache 16 0 14100 13M 8212 S 0.0 0.1 0:09 7 httpd
9674 apache 15 0 14472 14M 8160 S 0.0 0.1 0:05 0 httpd
9687 apache 16 0 14540 14M 8220 S 0.0 0.1 0:06 7 httpd
9708 apache 15 0 15020 14M 8116 S 0.0 0.1 0:06 3 httpd
9904 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 2 mysqld
10019 mysql 20 0 6832 6824 2200 S 0.0 0.0 0:00 3 mysqld
10034 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 3 mysqld
10436 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 3 mysqld
10723 root 15 0 3224 3224 2312 S 0.0 0.0 0:01 2 sendmail
10742 mysql 15 0 6832 6824 2200 S 0.0 0.0 0:00 3 mysqld
11654 root 15 0 688 688 604 S 0.0 0.0 0:00 5 procmail
11734 root 15 0 3316 3316 2316 S 0.0 0.0 0:00 7 sendmail
11778 root 20 0 3192 3192 2304 S 0.0 0.0 0:00 3 sendmail
11844 root 16 0 1896 1896 1576 S 0.0 0.0 0:00 5 ipop3d
11926 w491880 16 0 2100 2100 1764 D 0.0 0.0 0:00 0 ipop3d
11930 w311493 16 0 2088 2088 1752 D 0.0 0.0 0:00 1 ipop3d
11932 w310247 16 0 2088 2088 1752 D 0.0 0.0 0:00 5 ipop3d
11945 w302402 16 0 32652 31M 612 D 0.0 0.4 0:01 4 procmail
11957 root 21 0 960 960 812 S 0.0 0.0 0:00 5 xinetd
11960 root 21 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
11961 root 23 0 960 960 812 S 0.0 0.0 0:00 2 xinetd
11964 root 21 0 960 960 812 S 0.0 0.0 0:00 1 xinetd
11975 root 23 0 960 960 812 S 0.0 0.0 0:00 1 xinetd
11988 root 22 0 960 960 812 S 0.0 0.0 0:00 0 xinetd
12004 root 22 0 960 960 812 S 0.0 0.0 0:00 7 xinetd
12006 root 22 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12007 root 15 0 25536 24M 564 S 0.0 0.3 0:00 2 procmail
12016 root 22 0 960 960 812 S 0.0 0.0 0:00 1 xinetd
12019 root 22 0 960 960 812 S 0.0 0.0 0:00 2 xinetd
12021 root 22 0 960 960 812 S 0.0 0.0 0:00 3 xinetd
12024 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12025 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12027 root 23 0 960 960 812 S 0.0 0.0 0:00 7 xinetd
12037 root 22 0 960 960 812 S 0.0 0.0 0:00 3 xinetd
12054 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12056 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12060 root 23 0 960 960 812 S 0.0 0.0 0:00 7 xinetd
12066 root 24 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12068 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12073 root 22 0 960 960 812 S 0.0 0.0 0:00 0 xinetd
12074 root 22 0 960 960 812 S 0.0 0.0 0:00 5 xinetd
12076 root 22 0 960 960 812 S 0.0 0.0 0:00 1 xinetd
12103 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12108 root 16 0 3340 3340 2308 D 0.0 0.0 0:01 1 sendmail
12112 root 23 0 960 960 812 S 0.0 0.0 0:00 2 xinetd
12113 root 23 0 960 960 812 S 0.0 0.0 0:00 3 xinetd
12115 root 23 0 960 960 812 S 0.0 0.0 0:00 3 xinetd
12176 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12180 root 23 0 960 960 812 S 0.0 0.0 0:00 4 xinetd
12182 root 24 0 960 960 812 S 0.0 0.0 0:00 5 xinetd
12190 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12193 root 23 0 960 960 812 S 0.0 0.0 0:00 3 xinetd
12194 root 23 0 960 960 812 S 0.0 0.0 0:00 3 xinetd
12195 root 23 0 960 960 812 S 0.0 0.0 0:00 4 xinetd
12199 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12210 root 23 0 960 960 812 S 0.0 0.0 0:00 0 xinetd
12217 root 23 0 960 960 812 S 0.0 0.0 0:00 1 xinetd
12220 root 24 0 960 960 812 S 0.0 0.0 0:00 1 xinetd
12239 root 23 0 960 960 812 S 0.0 0.0 0:00 0 xinetd
12240 root 22 0 684 684 604 S 0.0 0.0 0:00 2 procmail
12244 root 23 0 960 960 812 S 0.0 0.0 0:00 7 xinetd
12260 root 23 0 960 960 812 S 0.0 0.0 0:00 3 xinetd
12263 root 19 0 3192 3192 2304 S 0.0 0.0 0:00 4 sendmail
12288 root 22 0 960 960 812 S 0.0 0.0 0:00 5 xinetd
12290 root 22 0 960 960 812 S 0.0 0.0 0:00 5 xinetd
12293 root 23 0 688 688 604 S 0.0 0.0 0:00 6 procmail
12296 root 20 0 3380 3380 2360 D 0.0 0.0 0:00 6 sendmail
12300 root 19 0 3320 3320 2308 D 0.0 0.0 0:00 1 sendmail
12301 w128027 23 0 2588 2588 1864 D 0.0 0.0 0:00 0 ipop3d
12304 root 22 0 3352 3352 2352 S 0.0 0.0 0:00 6 sendmail
12307 w384644 24 0 1864 1860 1436 D 0.0 0.0 0:00 1 imapd
12308 root 22 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12312 w317001 24 0 1580 1576 1316 D 0.0 0.0 0:00 7 imapd
12313 root 21 0 640 640 552 S 0.0 0.0 0:00 5 procmail
12314 root 20 0 3348 3348 2320 D 0.0 0.0 0:00 6 sendmail
12316 w184648 24 0 1752 1748 1472 D 0.0 0.0 0:00 6 imapd
12317 w393300 24 0 1636 1632 1368 D 0.0 0.0 0:00 0 imapd
12318 w327557 25 0 1624 1620 1364 D 0.0 0.0 0:00 2 imapd
12319 w306974 23 0 1608 1604 1348 D 0.0 0.0 0:00 3 imapd
12320 w300958 24 0 688 688 608 D 0.0 0.0 0:00 6 procmail
12321 w606967 24 0 2088 2088 1752 D 0.0 0.0 0:00 3 ipop3d
12322 w535633 24 0 1624 1620 1364 D 0.0 0.0 0:00 3 imapd
12323 w326742 23 0 2088 2088 1752 D 0.0 0.0 0:00 7 ipop3d
12324 w487260 24 0 2088 2088 1752 D 0.0 0.0 0:00 7 ipop3d
12326 root 15 0 648 648 556 S 0.0 0.0 0:00 4 crond
12328 root 23 0 516 516 432 S 0.0 0.0 0:00 3 process
12333 smmsp 23 0 2428 2420 1796 S 0.0 0.0 0:00 0 sendmail
12335 w547274 24 0 2088 2088 1752 D 0.0 0.0 0:00 6 ipop3d
12339 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12340 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12341 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12342 root 23 0 960 960 812 S 0.0 0.0 0:00 6 xinetd
12345 root 22 0 960 960 812 S 0.0 0.0 0:00 7 xinetd
12346 root 22 0 960 960 812 S 0.0 0.0 0:00 7 xinetd
12347 root 22 0 960 960 812 S 0.0 0.0 0:00 7 xinetd
12348 root 22 0 960 960 812 S 0.0 0.0 0:00 7 xinetd
12349 root 22 0 960 960 812 S 0.0 0.0 0:00 5 xinetd
12350 root 22 0 960 960 812 S 0.0 0.0 0:00 5 xinetd

USER is running:.....
1 instance(s) of >> STAT START TIME COMMAND

apache is running:.....
1 instance(s) of >> R 09:24 1:04 /usr/sbin/httpd
1 instance(s) of >> S 09:55 0:03 /usr/sbin/httpd
1 instance(s) of >> S 09:55 0:09 /usr/sbin/httpd
1 instance(s) of >> S 09:55 0:05 /usr/sbin/httpd
2 instance(s) of >> S 09:56 0:06 /usr/sbin/httpd
1 instance(s) of >> S 09:24 1:21 /usr/sbin/httpd
1 instance(s) of >> S 09:24 1:16 /usr/sbin/httpd
1 instance(s) of >> S 09:24 1:28 /usr/sbin/httpd
1 instance(s) of >> S 09:25 1:18 /usr/sbin/httpd
1 instance(s) of >> S 09:30 0:58 /usr/sbin/httpd
1 instance(s) of >> S 09:31 0:59 /usr/sbin/httpd
1 instance(s) of >> S 09:31 1:01 /usr/sbin/httpd
1 instance(s) of >> S 09:08 1:31 /usr/sbin/httpd
1 instance(s) of >> S 09:40 0:40 /usr/sbin/httpd
1 instance(s) of >> S 09:40 0:35 /usr/sbin/httpd
1 instance(s) of >> S 09:40 0:41 /usr/sbin/httpd
1 instance(s) of >> S 09:43 0:35 /usr/sbin/httpd
1 instance(s) of >> S 09:13 0:37 /usr/sbin/httpd
1 instance(s) of >> S 09:44 0:33 /usr/sbin/httpd
1 instance(s) of >> S 09:44 0:32 /usr/sbin/httpd
1 instance(s) of >> S 09:44 0:30 /usr/sbin/httpd

daemon is running:.....
1 instance(s) of >> S Dec07 0:02 /usr/sbin/atd

mailman is running:.....
1 instance(s) of >> S Dec07 0:00 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S Dec07 2:49 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S Dec07 2:51 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S Dec07 2:50 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S Dec07 2:51 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S Dec07 4:29 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S Dec07 3:23 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S Dec07 0:00 /usr/bin/python /var/mailman/bin
1 instance(s) of >> S Dec07 5:28 /usr/bin/python /var/mailman/bin

mysql is running:.....
1 instance(s) of >> S Dec07 0:26 /usr/libexec/mysqld --defaults-f

ntp is running:.....
1 instance(s) of >> SL Dec07 0:03 ntpd -U ntp -p /var/run/ntpd.pi

root is running:.....
1 instance(s) of >> D Dec07 91:02 sendmail: ./jB7BtbBM017560 from
1 instance(s) of >> D 07:54 11:40 sendmail: ./jB82ccwb004407 from
1 instance(s) of >> D 06:54 14:50 sendmail: ./jB7FL4lN024022 from
1 instance(s) of >> D 01:54 49:16 sendmail: ./jB4HXcZ2001462 from
1 instance(s) of >> D 09:54 0:41 sendmail: running queue: /var/sp
1 instance(s) of >> D 09:58 0:01 sendmail: jB8FwiNJ012108 mcvs3.o
1 instance(s) of >> D 09:58 0:00 sendmail: ./jB8FwiNF012108 from
1 instance(s) of >> D 09:58 0:00 sendmail: jB8FwvhN012300 mcvs2.o
1 instance(s) of >> D 09:58 0:00 sendmail: ./jB8FwiNH012108 from
1 instance(s) of >> D 09:59 0:00 /bin/ksh /ocean/sysadm/rejohn/pr
1 instance(s) of >> D Dec07 76:39 sendmail: ./jB2EVMP0003572 from
1 instance(s) of >> DW Dec07 59:29 [kjournald]
1 instance(s) of >> DW Dec07 3:45 [kjournald]
1 instance(s) of >> R 09:59 0:00 /bin/ps auxwww
1 instance(s) of >> S Dec07 1:13 init
1 instance(s) of >> S Dec07 1:23 syslogd -m 0
1 instance(s) of >> S Dec07 0:01 klogd -x
1 instance(s) of >> S Dec07 0:31 irqbalance
1 instance(s) of >> S Dec07 0:00 /usr/sbin/sshd
1 instance(s) of >> S Dec07 5:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S Dec07 0:00 /bin/sh /usr/bin/safe_mysqld --d
1 instance(s) of >> S Dec07 3:54 sendmail: accepting connections
1 instance(s) of >> S Dec07 1:44 /opt/tivoli/tsm/client/ba/bin/ds
1 instance(s) of >> S Dec07 0:00 gpm -t imps2 -m /dev/mouse
1 instance(s) of >> S Dec07 0:03 /usr/sbin/httpd
1 instance(s) of >> S Dec07 0:00 /usr/sbin/radiusd -y
1 instance(s) of >> S Dec07 0:04 crond
1 instance(s) of >> S Dec07 0:00 rhnsd --interval 240
1 instance(s) of >> S 08:50 0:00 ipop3d
1 instance(s) of >> S Dec07 0:00 ipop3d
1 instance(s) of >> S 09:52 0:00 ipop3d
1 instance(s) of >> S 08:54 5:24 sendmail: ./jB8EEBiw001465 from
1 instance(s) of >> S 09:57 0:01 sendmail: server mcvs3.otr.usm.e
1 instance(s) of >> S 09:58 0:00 procmail -f MAILER-DAEMON@mail.u
1 instance(s) of >> S 09:58 0:00 sendmail: ./jB8FwE3B011733 from
1 instance(s) of >> S 09:58 0:00 sendmail: server mcvs2.otr.usm.e
1 instance(s) of >> S 09:58 0:00 ipop3d
39 instance(s) of >> S 09:58 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 09:58 0:00 procmail -f MAILER-DAEMON@mail.u
2 instance(s) of >> S 09:58 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 09:58 0:00 sendmail: server mcvs1.otr.usm.e
2 instance(s) of >> S 09:58 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 09:58 0:00 procmail -f MAILER-DAEMON@mail.u
1 instance(s) of >> S 09:58 0:00 sendmail: ./jB8FwvhL012300 from
1 instance(s) of >> S 09:58 0:00 xinetd -stayalive -pidfile /var/
1 instance(s) of >> S 09:58 0:00 procmail -f RollandRoberts@aol.c
1 instance(s) of >> S 09:59 0:00 CROND
1 instance(s) of >> S 09:59 0:00 /bin/ksh /ocean/sysadm/rejohn/pr
11 instance(s) of >> S 09:59 0:00 xinetd -stayalive -pidfile /var/
2 instance(s) of >> S 09:59 0:00 ipop3d
1 instance(s) of >> S 04:54 34:10 sendmail: ./jB6Dk6MV020522 from
1 instance(s) of >> S 02:54 43:24 sendmail: ./jB5KFjID008700 from
1 instance(s) of >> S Dec07 0:00 ipop3d
1 instance(s) of >> S 09:35 0:00 sendmail: server mcvs3.otr.usm.e
1 instance(s) of >> S 05:54 21:42 sendmail: ./jB6MoDhB002146 from
1 instance(s) of >> S Dec07 85:02 sendmail: ./jB7E94NZ028956 from
1 instance(s) of >> S 03:54 36:31 sendmail: ./jB6GWGEh028967 from
1 instance(s) of >> S 00:54 61:43 sendmail: ./jB30VMhZ025360 from
1 instance(s) of >> S< Dec07 0:01 ncftpd -d /usr/local/etc/ncftpd
3 instance(s) of >> S< Dec07 0:00 ncftpd -d /usr/local/etc/ncftpd
2 instance(s) of >> S< 09:54 0:00 ncftpd -d /usr/local/etc/ncftpd
1 instance(s) of >> SN Dec07 0:02 ncftpd -d /usr/local/etc/ncftpd
1 instance(s) of >> SW Dec07 0:00 [migration/0]
1 instance(s) of >> SW Dec07 0:00 [migration/1]
1 instance(s) of >> SW Dec07 0:00 [migration/2]
1 instance(s) of >> SW Dec07 0:00 [migration/3]
1 instance(s) of >> SW Dec07 0:00 [migration/4]
1 instance(s) of >> SW Dec07 0:00 [migration/5]
1 instance(s) of >> SW Dec07 0:00 [migration/6]
1 instance(s) of >> SW Dec07 0:00 [migration/7]
1 instance(s) of >> SW Dec07 0:00 [keventd]
1 instance(s) of >> SW Dec07 0:36 [kswapd]
1 instance(s) of >> SW Dec07 0:12 [kscand]
1 instance(s) of >> SW Dec07 0:00 [bdflush]
1 instance(s) of >> SW Dec07 0:31 [kupdated]
1 instance(s) of >> SW Dec07 0:00 [mdrecoveryd]
1 instance(s) of >> SW Dec07 0:00 [ahc_dv_0]
1 instance(s) of >> SW Dec07 0:00 [ahc_dv_1]
1 instance(s) of >> SW Dec07 0:00 [scsi_eh_0]
1 instance(s) of >> SW Dec07 0:00 [scsi_eh_1]
1 instance(s) of >> SW Dec07 0:00 [scsi_eh_2]
1 instance(s) of >> SW Dec07 0:30 [kjournald]
1 instance(s) of >> SW Dec07 0:00 [khubd]
1 instance(s) of >> SW Dec07 0:00 [kjournald]
1 instance(s) of >> SW Dec07 0:28 [kjournald]
1 instance(s) of >> SW Dec07 0:46 [kjournald]
1 instance(s) of >> SWN Dec07 0:00 [ksoftirqd/0]
1 instance(s) of >> SWN Dec07 0:00 [ksoftirqd/1]
1 instance(s) of >> SWN Dec07 0:00 [ksoftirqd/2]
1 instance(s) of >> SWN Dec07 0:00 [ksoftirqd/3]
1 instance(s) of >> SWN Dec07 0:00 [ksoftirqd/4]
1 instance(s) of >> SWN Dec07 0:00 [ksoftirqd/5]
1 instance(s) of >> SWN Dec07 0:00 [ksoftirqd/6]
1 instance(s) of >> SWN Dec07 0:00 [ksoftirqd/7]
1 instance(s) of >> S 09:58 0:00 procmail -f MAILER-DAEMON@mail.u
1 instance(s) of >> S Dec07 0:00 /sbin/mingetty tty1
1 instance(s) of >> S Dec07 0:00 /sbin/mingetty tty2
1 instance(s) of >> S Dec07 0:00 /sbin/mingetty tty3
1 instance(s) of >> S Dec07 0:00 /sbin/mingetty tty4
1 instance(s) of >> S Dec07 0:00 /sbin/mingetty tty5
1 instance(s) of >> S Dec07 0:00 /sbin/mingetty tty6

smmsp is running:.....
1 instance(s) of >> S Dec07 0:00 sendmail: Queue runner@01:00:00
1 instance(s) of >> S 09:59 0:00 /usr/sbin/sendmail -FCronDaemon

w105392 is running:.....
1 instance(s) of >> S 09:46 0:00 imapd

w124825 is running:.....
1 instance(s) of >> S Dec07 0:27 imapd

w128027 is running:.....
1 instance(s) of >> D 09:58 0:00 ipop3d

w137302 is running:.....
1 instance(s) of >> D 09:58 0:00 ipop3d

w184648 is running:.....
1 instance(s) of >> D 09:59 0:00 imapd

w188556 is running:.....
1 instance(s) of >> D 09:59 0:00 ipop3d

w300296 is running:.....
1 instance(s) of >> S 08:07 0:01 imapd

w300957 is running:.....
1 instance(s) of >> D 09:59 0:00 imapd

w300958 is running:.....
1 instance(s) of >> D 09:58 0:00 ipop3d
1 instance(s) of >> D 09:59 0:00 procmail -f neljmlywe@xpresscom.

w301176 is running:.....
1 instance(s) of >> D 09:59 0:00 ipop3d

w302402 is running:.....
1 instance(s) of >> D 09:58 0:01 procmail -f MAILER-DAEMON@mail.u

w302651 is running:.....
1 instance(s) of >> S 09:38 0:02 imapd

w303825 is running:.....
1 instance(s) of >> S 09:40 0:01 imapd

w306038 is running:.....
1 instance(s) of >> S 08:52 0:01 imapd
1 instance(s) of >> S 08:41 0:02 imapd

w306777 is running:.....
1 instance(s) of >> D 09:59 0:00 ipop3d

w306974 is running:.....
1 instance(s) of >> D 09:59 0:00 imapd

w310247 is running:.....
1 instance(s) of >> D 09:58 0:00 ipop3d

w311493 is running:.....
1 instance(s) of >> D 09:58 0:00 ipop3d

w317001 is running:.....
1 instance(s) of >> D 09:59 0:00 imapd

w326742 is running:.....
1 instance(s) of >> D 09:59 0:00 ipop3d

w327557 is running:.....
1 instance(s) of >> D 09:59 0:00 imapd

w384644 is running:.....
1 instance(s) of >> D 09:58 0:00 imapd

w393300 is running:.....
1 instance(s) of >> D 09:59 0:00 imapd

w487260 is running:.....
1 instance(s) of >> D 09:59 0:00 ipop3d

w487907 is running:.....
1 instance(s) of >> S 09:34 0:03 ipop3d

w491880 is running:.....
1 instance(s) of >> D 09:58 0:00 ipop3d

w492038 is running:.....
1 instance(s) of >> S 08:31 0:01 imapd

w535633 is running:.....
1 instance(s) of >> D 09:59 0:00 imapd

w547274 is running:.....
1 instance(s) of >> D 09:59 0:00 ipop3d

w606967 is running:.....
netstat.............
tcp 102 0 131.95.82.42:995 131.95.113.105:1139 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.204.24:1212 ESTABLISHED
tcp 0 0 131.95.82.42:443 131.95.198.82:3056 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.20.196:1395 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.198:1215 ESTABLISHED
tcp 102 0 131.95.82.42:995 198.49.215.87:1488 ESTABLISHED
tcp 0 0 127.0.0.1:51663 127.0.0.1:143 ESTABLISHED
tcp 0 0 127.0.0.1:51666 127.0.0.1:143 ESTABLISHED
tcp 0 0 127.0.0.1:51679 127.0.0.1:143 ESTABLISHED
tcp 0 25 127.0.0.1:51677 127.0.0.1:143 ESTABLISHED
tcp 0 0 127.0.0.1:51675 127.0.0.1:143 ESTABLISHED
tcp 0 0 127.0.0.1:51638 127.0.0.1:143 ESTABLISHED
tcp 0 0 127.0.0.1:51642 127.0.0.1:143 ESTABLISHED
tcp 0 0 131.95.82.42:443 131.95.121.81:1818 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.240.116:1457 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.237.49:1512 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.194:1478 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.237.89:2223 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.198.94:1414 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.85.17:2386 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.175.111:1087 ESTABLISHED
tcp 0 51 131.95.82.42:995 131.95.169.113:1068 ESTABLISHED
tcp 0 0 131.95.82.42:443 131.95.177.213:1137 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.164.152:2329 ESTABLISHED
tcp 0 0 127.0.0.1:143 127.0.0.1:51638 ESTABLISHED
tcp 0 0 127.0.0.1:143 127.0.0.1:51642 ESTABLISHED
tcp 0 0 131.95.82.42:995 69.138.104.240:33138 ESTABLISHED
tcp 120 0 131.95.82.42:995 131.95.213.121:1131 ESTABLISHED
tcp 102 0 131.95.82.42:995 68.63.236.4:2637 ESTABLISHED
tcp 0 0 127.0.0.1:143 127.0.0.1:51663 ESTABLISHED
tcp 0 0 127.0.0.1:143 127.0.0.1:51666 ESTABLISHED
tcp 0 76 127.0.0.1:143 127.0.0.1:51675 ESTABLISHED
tcp 0 0 127.0.0.1:143 127.0.0.1:51677 ESTABLISHED
tcp 0 0 127.0.0.1:143 127.0.0.1:51679 ESTABLISHED
tcp 78 0 131.95.82.42:995 70.155.212.220:54696 ESTABLISHED
tcp 0 0 131.95.82.42:995 69.138.104.240:33066 ESTABLISHED
tcp 0 0 131.95.82.42:993 208.148.211.119:1254 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.231.130:4832 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.197.128:4301 ESTABLISHED
tcp 0 111 131.95.82.42:995 131.95.242.163:1497 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.237.80:1177 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.211:3837 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.89.12:1370 ESTABLISHED
tcp 78 0 131.95.82.42:995 69.138.103.202:1138 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.26.27:4073 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.55:43394 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.240.93:1398 ESTABLISHED
tcp 124 0 131.95.82.42:995 131.95.170.55:1372 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.213.184:1363 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.229.40:1546 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.192.161:1328 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.242.139:1061 ESTABLISHED
tcp 102 0 131.95.82.42:995 68.63.230.175:3952 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.55:43203 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.213.12:1367 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.204.105:3362 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.55:40673 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.240.101:1296 ESTABLISHED
tcp 78 0 131.95.82.42:995 131.95.204.162:1769 ESTABLISHED
tcp 126 0 131.95.82.42:995 209.58.248.30:37620 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.17.67:60978 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.242.20:3516 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.59.229:1133 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.180:2002 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.100.130:3417 ESTABLISHED
tcp 120 0 131.95.82.42:995 131.95.100.118:1577 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.69.11:1121 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.213.175:1448 ESTABLISHED
tcp 0 0 131.95.82.42:995 198.49.215.149:1520 ESTABLISHED
tcp 0 20440 131.95.82.42:995 4.231.83.149:3741 ESTABLISHED
tcp 0 0 131.95.82.42:993 131.95.50.148:1106 ESTABLISHED
tcp 0 1735 131.95.82.42:995 131.95.26.16:1287 ESTABLISHED
tcp 0 0 131.95.82.42:443 131.95.197.111:51271 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.237.183:1150 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.121.87:1433 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.121.87:1435 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.121.87:1434 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.56:39627 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.150.212:4134 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.213.51:1403 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.22.181:4911 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.241.46:2231 ESTABLISHED
tcp 0 146 131.95.82.42:443 70.152.116.107:50615 ESTABLISHED
tcp 0 0 131.95.82.42:443 70.152.116.107:50614 ESTABLISHED
tcp 0 146 131.95.82.42:443 70.152.116.107:50617 ESTABLISHED
tcp 0 146 131.95.82.42:443 70.152.116.107:50616 ESTABLISHED
tcp 0 146 131.95.82.42:443 70.152.116.107:50621 ESTABLISHED
tcp 0 0 131.95.82.42:993 129.173.105.114:57743 ESTABLISHED
tcp 0 146 131.95.82.42:443 70.152.116.107:50620 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.231.164:3559 ESTABLISHED
tcp 0 0 131.95.82.42:443 68.63.230.213:1763 ESTABLISHED
tcp 0 0 131.95.82.42:443 131.95.218.221:1206 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.174.101:1495 ESTABLISHED
tcp 0 0 131.95.82.42:25 131.95.82.57:51555 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.204.23:1296 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.37:2107 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.85.50:1237 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.165.62:1591 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.167.112:1093 ESTABLISHED
tcp 0 0 131.95.82.42:995 131.95.42.25:3000 ESTABLISHED
tcp 102 0 131.95.82.42:995 131.95.22.169:4555 ESTABLISHED
tcp 148 0 131.95.82.42:995 131.95.218.109:2622 ESTABLISHED
tcp 0 0 131.95.82.42:443 131.95.219.32:1654 ESTABLISHED
udp 0 0 131.95.82.42:50055 131.95.84.2:53 ESTABLISHED
udp 0 0 131.95.82.42:50059 198.49.215.21:53 ESTABLISHED
udp 0 0 131.95.82.42:50064 131.95.84.2:53 ESTABLISHED
udp 0 0 131.95.82.42:50067 131.95.242.10:53 ESTABLISHED
udp 0 0 131.95.82.42:50073 198.49.215.21:53 ESTABLISHED
udp 0 0 131.95.82.42:50078 131.95.242.10:53 ESTABLISHED
udp 0 0 131.95.82.42:50139 131.95.84.2:53 ESTABLISHED
udp 0 0 131.95.82.42:50150 131.95.84.2:53 ESTABLISHED
udp 0 0 131.95.82.42:50151 198.49.215.21:53 ESTABLISHED
udp 0 0 131.95.82.42:50152 131.95.242.10:53 ESTABLISHED
udp 0 0 131.95.82.42:50163 198.49.215.21:53 ESTABLISHED

End of program: proclist

 

[mrballcb]

That's not exactly what happens to us with Apache. For us something causes a single apache process to get really large very quickly and it grabs all available memory and the kernel starts killing off all the other processes.

Process status D is what is causing your problem. D means "waiting on I/O". It means a userland process has told the kernel to either read or write something to a filesystem and it's waiting for either the answer or a write confirmation. Normally it is very fast, nearly instantaneous. But at this exact time in the morning, something happens that causes the answers to be very slow (or it goes away for a few minutes).

No single process grows large enough to consume lots of memory, so that's why we know you're not seeing the same thing as I described in the first paragraph above.

Something is happening that is making access to your filesystem or network filesystem go away for a while. Is the home directories on an NFS server? Is the mailspool on an NFS server? If the NFS server goes away for a while, all processes that are accessing things. By any chance is a backup of this machine occurring at the same time? A backup of the NFS server (NDMP dumps hit NFS fileservers really hard, but usually not hard enough to make NFS clients back up I/O far enough to crash them).

I wonder what mrtg graphs of this machine and the server would look like around this time.

If you don't have an NFS server and it's only a local LPAR, then I'd have to say it was something occurring on the host OS itself. But I don't think you have a mainframe, so that seems unlikely too.

 

[rejohn]

all filesystems are local to the server.

 

[mrballcb]

If there are no network fs's then I'm not sure what could be causing all of those processes to go to the D state at the same time. Something on your system is happening, you think it's sendmail related, I'm not so sure. But either way, it's happening at the same time each night.

What cron jobs are there? What times are they set to run? Include user cron jobs. Is there any correlation to the time that the server locks up?

 

[rejohn]

lately the freezes have not been at the same time. I have a IBM PCI ServeRAID 7.10.18 Build 731 <ServeRAID 6M> cabinet internal to the server, I will flash to the latest firmware available for the ServeRAID.
-------------------------------------------------------------
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.31440 installed on Fri Aug 20 16:10:55 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
#
# Runs SCAMP administrator on ocean... -JD 04/01/98
#
* * * * * /usr/sysadm/bin/scamp-account-run-oracle.pl >> /var/log/scamplog 2>> /var/log/scamplog.err


# Daily scan of mcvs server for viruses (Required MailWatch)
#0 0 * * * /usr/sysadm/bin/daily_infected_machine_scan.pl

#
# Scan mailman for proper permissions
#
#0 1 * * * /var/mailman/bin/check_perms

# Run list compilations as per George Booth
0 * * * * /ocean/fs_mailman/cron/do_it_itechall
2 * * * * /ocean/fs_mailman/cron/do_it_gcall
4 * * * * /ocean/fs_mailman/cron/do_it_students

#
# Check password for expiration, and send notices
#
0 0 * * * /usr/sysadm/bin/checkpwdexp

# RUN BY HAND IF THERE IS A NETWORK OUTAGE
# --> old automation do not run */5 * * * * /usr/sysadm/bin/do_isp_run
# Stopped processing new users as per Isaac Johnston's request on 2/17/2005
#*/5 * * * * /root/isp_automation/isp_automation.sh 2>> /root/isp_automation/isp_automation.log
#*/5 * * * * /sbin/service mailman restart > /dev/null

# ISP usage report automation
#0 6,18 * * * /usr/sysadm/bin/eagleconn_usage/isp_usage.sh

# Web Mail Database Archive
0 17 * * * /usr/sysadm/bin/hordeOceanBackup

# Scan ocean for old processes that should be removed
#0 7 * * * /ocean/sysadm/falgout/ocean_process_scan.pl

# clock sync with time server to time.nist.gov
#0 * * * * /usr/sbin/ntpdate 192.43.244.18 > /dev/null

# send out shadow file if a password changes
*/5 * * * * /root/shadow_copy.pl

# pull mail stats on the first day of each month
1 0 1 * * /root/mailstats.sh

# report: quota usage for all filesystems
#0 0 * * * /root/quota_report.sh
#3 12 * * * /root/quota_report.sh

# quotacheck: keeps the system in balance otherwise the quota system will slowly be corrupted
#0 3 * * * /sbin/quotacheck -avfm
0 3 * * * /sbin/quotacheck -avug


#Checking process and memory every 5 mins
*/2 * * * * /ocean/sysadm/rejohn/process