root 100% full

[khalidaaa]

Hi AIXers,

I have a very strange problem! my root filesystem get 100% full and i can't see any thing unusual inside it! it happens for 3 times and i used to increase it! now it is 1G and still i got the same problem again!

I'm running AIX 5.2. any help would be appreciated.

Regards,
Khalid

 

[khalidaaa]

The %Iused was 81% and now after i increased the root filesystem by 250MB it becomes 4%!!!

Inodes used!?!? what is happening?

Regards,
Khalid

 

[mrn]

What file system type are you using?
If jfs2 are you using inline logs?

Was just thinking maybe when you increase the space the log is getting flushed and releasing the inodes.



Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[khalidaaa]

Thanks for the reply Mike.

I'm using JFS2 with JFS2log! not inline.

Then how should i come over this problem! I stil expect it to come soon! as it reoccured three times so far!

Regards,
Khalid

 

[mrn]

Trying using lsof to see if anything is writing to /

Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[khalidaaa]

That's what i thought as well! I have lsof installed on the machine but i'm still reading thru its man pages to know how to use it!

Regards,
Khalid

 

[sbrews]

The quick and easy way to use lsof would be:

lsof /

it will list all files that are currently open.

Add a little color to your PUTTY terminal: faq52-6627

 

[khalidaaa]

Thanks sbrews. But that was really really long list that i couldn't figure out what to look for!

any idea?

Regards,
Khalid

 

[p5wizard]

I'd try

find / -xdev -mtime -1 -ls

to see what files are growing inside the root filesystem...

Also, IMHO you need to be pretty recent on your MLs/TLs and SPs with AIX 5.2 to use JFS2 without too much trouble. I only use JFS2 on AIX 5.3 .


HTH,

p5wizard

 

[TSch]

I know it's not the best guess, but you could search for all *.log files in the root filesystem and have a look at them. Continuously growing Logfiles often proove to be the cause for such problems.

 

[p5wizard]

Also, you might want to check if an open file of some sorts isn't hidden away by a mounted filesystem. A situation like that would limit find's capability to locate that file and would still allow an application to fill up the parent filesystem in question...

/log/appl.log <- open file in / FS
crfs -asize=1G /log <- create new filesystem
mount /log <- mount new FS, while file /log/appl.log is still open
result: appl.log file is still growing inside / FS and possibly filling it up...


HTH,

p5wizard

 

[mrn]

Also watchout for a file being written to (open) which then gets deleted, will not show up but still use disk space until closed, next time it fills up try a reboot if you get your space back, this is the avenue to look down.

Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[khalidaaa]

Thanks alot for your input guys.

After i increased the / i don't see the problem happening again!

And by the way this problem happened and persist after the reboot! but now i don't see any problem.

oslevel -s
5200-08-00

I will try to monitor this for a while and i will let you know once it happens again.

Thanks again.

Regards,
Khalid

 

[McleanWJM]

Yeah the find -xdev trick is the first thing I do, If you still cant find the file check lsof to see if someone remove a log and the file handle is still open .

 

[khalidaaa]

Thanks McleanWJM for the addition

As I said i won't be able to be sure of what these commands do until i have the problem again which fortunately is not there for now! so i will lay low for a while till it comes back again!

Regards,
Khalid

 

[p5wizard]

Still, you can run the "find / -xdev -mtime -1 -ls" (or -2 or -3 ...) to see any recent file in / filesystem, other than some legitimate smit log & script files that output shouldn't be a heck of a long listing...

I'd also look for an erroneous ">>/dev/nul" (note single "l") in some scripts that would silently collect garbage on disk in / filesystem instead of dumping it to the "real" bit bucket /dev/null.

Of course if some other frequently run script has ">/dev/nul", that would empty the garbage bin for you, and would limit the amount of damage done. Could be that this is what is happening right now...


HTH,

p5wizard