File system full, swap space limit exceed !!

[babeo]

Hi

I get the warning message from my server. I would like to know what exactly it means, what is the cause and how to fix it

Jul 19 05:15:01 ondaily unix: WARNING: /tmp: File system full, swap space limit exceeded

In fact, I check the space on my server and it confuses me more:

ondaily:/tmp>date
Wednesday July 18 09:11:53 EDT 2001
ondaily:/tmp>df -k
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d0 3728870 1140060 2551522 31% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
/dev/md/dsk/d2 4124422 2114885 1968293 52% /ondaily
/dev/md/dsk/d3 8705501 5172770 3445676 61% /ondaily/archive
swap 125512 47392 78120 38% /tmp
testserverndaily/test 4071192 2894320 1136160 72% /test

ondaily:/tmp>date
Thursday July 19 09:39:55 EDT 2001
ondaily:/tmp>df -k
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d0 3728870 1140695 2550887 31% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
/dev/md/dsk/d2 4124422 2220968 1862210 55% /ondaily
/dev/md/dsk/d3 8705501 5662262 2956184 66% /ondaily/archive
swap 524552 47400 477152 10% /tmp
testserverndaily/test 4071192 2894320 1136160 72% /test

Another question: I have a map directory from another server onto my server, could be the problem on that server (testserver) affect or display the message on my server too? (my experience is I usually see the message from that server displays on my server when it is reboot.)
Thanks

 

[grega]

Just a few thoughts ...

First off, it looks like your swap space has been increased from 125Mb to 524Mb between the 18th and 19th.

Are you running any jobs at that time of the morning (5:15am) which would make use of /tmp?

I wouldn't think that your remote mounted directory has any effect.

Greg.

 

[babeo]

Hi Greg

I think the same thing about maybe someone has a cron job running at that time, but not sure, that is why I confuse when df to check space !!. Is there anyway to check ALREADY finished the cron job (I can check the routine cron job by crontab -l, but don't know how to check the cron job using "at" command, once it finished)

However, what is the solution for this problem if I keep getting the message? Also any affectation because of this? This is once time set up and is a production system 24x7, I can not take it off line for longer than 2 hours, and I am afraid of messing up the configuration or system files.

I think I read about this problem somewhere else in this forum, and someone suggests to increase the space of /tmp for swapping, and more, but I don't remember and could not find where that message is. Any suggestion?
Thanks

 

[grega]

I personally think 500Mb of swap space is plenty.

I would try and find out if there are any jobs running at that time ...

- check with your sys admin to find out all cron & at jobs which are scheduled to run. Also find out who is allowed to run cron jobs.

- If the error occurs at roughly the same time each morning (say 5:15), you could schedule a script of your own to run every 30 seconds between 5:10 and 5:20 to log the output of df -k and ps -ef so you can check /tmp usage & running processes.

Just a few ideas.

Greg.

 

[babeo]

Thanks
I think your suggestion is good, but I may have jam with system emails of df -k and ps -ef . I will try anyway.

 

[Guest_imported]

How is your system doing on memory? Run vmstat 5 and check the swap rate(sr). If you are paging you will be using up swap space.

 

[mikeclark]

If you find a few jobs running at the same time that use a lot of swap space, see if you can stagger them so they are not all running at the same time.