Inodes not being released !! 4

[joedyq]

We have a situation where the filesystem is showing as full, in actuality its not !! Inodes have not been released post large file deletion . Is there anyway to gwt this space back without rebooting the server . I have tried resizing the filesystem in the hope that the action might do the job !!

J

 

[MoshiachNow]

Hi,

Please post result of "df ". "Long live king Moshiach !"

 

[joedyq]

the out put of df is as follows:

5013504 2016 100% 4853 1% /home

The output of du -sk /home is .

107772 /home

 

[MoshiachNow]

Looks like your FS is really full.

Inodes are used only 1%.

Can run the following to display 10 biggest directories on the FS by size:

du -a <filesystem> | sort -n -r | head -10 &quot;Long live king Moshiach !&quot;

 

[joedyq]

I have done all of that , the du ouput shows you something different from the output of df . Basically there are no large files here , not anymore anyway . I have used a script to also get the total of all files in the filesystem and it reflects the output of the du command .

J

 

[MoshiachNow]

Ok.

Try unmounting the FS and running &quot;fsck -y&quot;,then mounting it back. &quot;Long live king Moshiach !&quot;

 

[joedyq]

Thank you , I was hoping to avoid that aswell !! Being a very busy production system , this is just as intrusive as a server down .

J

 

[gheist]

fsck helps as extreme solutions - if you cannot umount filesystem a 'lsof' of 'fuser' utilities might be of help (lsof is not IBM's - you shoud download it separately)

like
fuser /mountpoint
or
lsof | grep /dev/device
and kill offending processes

basically you cannot delete an open file - you lose it out of your sight but another user can still write to it using inodes,
so beter .. cp logfile newfile ; >logfile

 

[Yegolev]

It has been my experience that just a umount/mount will do the trick, in the case of df not reflecting actual fs usage after large file deletion.

 

[gheist]

how do you umount busy filesystem safely?

 

[gheist]

how do you umount busy filesystems safely?

 

[Yegolev]

Safely? Hehehe. There is no safely, you need to get users out of the dir and kill the processes. Use fuser to see which ones to kill. Sometimes just killing the right process can make df start to work.

 

[MoshiachNow]

Another option to get a 100% full FS working till next reboot/umount is adding couple of free partiotions (if available..) to the particular FS. &quot;Long live king Moshiach !&quot;

 

[joedyq]

Thank you for the suggestions , I will try the fuser option again , but unfortunately , this system has far too many actual unix users !!

 

[Yegolev]

levw,

Maybe a synclvodm or something like that will work, too? This has not happened in a while to me.

As for the actual thread, if no users are complaining then do not worry about it. If a tree falls in the forest and no one hears it... it doesn't go in the outage reports. Just fix it during next maintenance window. Or, if they complain just say you need the machine for an hour and just reboot.

 

[avadh]

Hello,
On what basis you can say that Inodes are not being released. What proof or indication that you have that say inodes are not being released.
What are you trying to do.

 

[gheist]

dd if=/dev/zero of=datafile & rm datafile

and watch your filesystem filling

 

[avadh]

Hello,
That does not answer my question. Need a better explanation that that command. That is a special file and it has nothing to do with file system.
This seems like a hypothetical problem and not a real problem in code.

 

[gheist]

ok - my explanation

dd reads some data from /dev/zero ( but can read from anywhere else ) and writes it to some normal file, then that file is removed, but process still writes on open inode ( althogh file is gone)

 

[gheist]

not an aix problem - works on most unices

dd if=/dev/zero of=datafile & sleep 5 ; rm datafile

for better results